大文件读取问题 C++/MFC

不 Seek 直接先读到中间，
char *pTextBuffer = new char【1024*1024*512】；
m_StdioFile.Read(pTextBuffer, 1024*1024*512);看看速度如何？

1：试下fopen,fseek,fread。MFC的文件操作比纯C的方式慢很多。
2：1L说的

不懂这就是一个学习的机会，详细点的资料可以看windows核心编程，专门有一章讲内存映射文件的。另百度下也有很多资料懒得贴链接了

这里有一帖与你遇到的问题一样，可参照
http://bbs.csdn.net/topics/370020918

Seek 只是做标记
读的时候，才前进。
char *pTextBuffer = new char【1024*1024】；// 1 M
m_StdioFile.Read(pTextBuffer, 1024*1024);
看是不是快了

据说 “从理论上讲内存映射文件比直接读文件要快，因为减少了I/O request packets(IRP)生成。”

#include <sys\stat.h>
#include <io.h>
#include <fcntl.h>
#include <share.h>
#include <stdlib.h>
#include <stdio.h>
#include <conio.h>
#include <string.h>
#define MAX_CLU_BYTES 65536
FILE *fo;
int fh;
__int64 offs,offs1;
__int64 rvi64;
int rv,wrv;
unsigned char buf[MAX_CLU_BYTES];
char ofn[_MAX_PATH];
char offstr[80];
void strcpybutcomma(char *t,char *s) {
    char c;    while (1) {
        c = *s++;
        if (','!=c) *t++ = c;
        if (0==c) break;
    }
}
void main(int argc,char **argv) {
    if (argc<3) {
        printf("Copy File Tail.\n");
        printf("Usage:\n");
        printf("    cft filename.ext offset_begin[-offset_end]\n");
        printf("Copy filename.ext offset_begin[-offset_end] to offset_begin[-offset_end]-filename.ext\n");
        printf("Note: Byte at offset_end is NOT included.\n");
        printf("Example:\n");
        printf("    cft abc.rar 12345\n");
        printf("Copy abc.rar offset 12345-end to 12345-abc.rar\n");
        printf("    cft abc.rar 123-12345\n");
        printf("Copy abc.rar offset 123-12345 to 123-12345-abc.rar\n");
        printf("    cft abc.rar 0xAB-0xCD\n");
        printf("Copy abc.rar offset 0xAB-0xCD to 0xAB-0xCD-abc.rar\n");
        return;
    }
    strcpybutcomma(offstr,argv[2]);
    rv=sscanf(offstr,"%I64i-%I64i",&offs,&offs1);
    if (rv==0) {
        printf("offset %s is not number\n",argv[2]);
        return;
    }
    fh=_sopen(argv[1],_O_BINARY|_O_RDONLY|_O_RANDOM,_SH_DENYWR);
    if (fh==-1) {
        printf("_sopen %s errno=%d\n",argv[1],errno);
        return;
    }
    if (rv==1) {
        offs1=_filelengthi64(fh);
        if (offs1==-1i64) {
            printf("%I64=_filelengthi64 errno=%d\n",offs1,errno);
            _close(fh);
            return;
        }
    } else {//rv==2
        if (offs1<offs) {
            printf("%s offset_begin>offset_end error\n",argv[2]);
            _close(fh);
            return;
        }
    }
    rvi64=_lseeki64(fh,offs,SEEK_SET);
    if (rvi64!=offs) {
        printf("%I64u=_lseeki64 %I64u errno=%d\n",rvi64,offs,errno);
        _close(fh);
        return;
    }
    sprintf(ofn,"%s-",offstr);
    strcat(ofn,argv[1]);
    fo=fopen(ofn,"wb");
    if (fo==NULL) {
        _close(fh);
        printf("fopen %s error\n",ofn);
        return;
    }
    cprintf("\n%I64u\r",offs);
    while (1) {
        rv=_read(fh,buf,(unsigned int)__min(offs1-offs,MAX_CLU_BYTES));
        if (rv==0) break;//
        if (rv<0) {
            fclose(fo);
            _close(fh);
            printf("_read %s offset %I64u error\n",argv[1],offs);
            return;
        }
        wrv=fwrite(buf,1,rv,fo);
        if (wrv!=rv) {
            fclose(fo);
            _close(fh);
            printf("fwrite %s error\n",ofn);
            return;
        } else {
            offs+=rv;
            cprintf("%I64u\r",offs);
            if (offs>=offs1) break;//
        }
    }
    fclose(fo);
    _close(fh);
    printf("Copy %s offset %s to %s OK.\n",argv[1],argv[2],ofn);
}

记不得哪位C++大牛在哪本学习C++的书的前言里面说过
“用C语言1000行源码能完成的工作千万不要用C++重写！”

这个问题有这么复杂吗？！
lz去查查msdn，seek函数只是移动了文件指针，并没有实际读文件，所以它能瞬间完成。
至于解决方案，试试内存映射喽，这个可以问百度。

那你就更错了，seek并没有移动文件的指针，只是在哪个位置做了一个标识。要是文件的指针移动了的话，读文件就不会出现这个问题了，我要的是解决的方法，不是跟我说Seek的问题。

我可以说原因并非楼上分析的那样吗？
我工作中遇到的文件1.5GB都算小的了，正常的文件都是以10G为单位的，但并没有因此用到内存印射，普通方式seek，再read完全没有问题。
我也按楼主的方式测试过了，文件大小2GB，seek位置也一样，读取大小也一样，很快的完成。
既然提供了seek，那么seek肯定是有用的，要不然seek有什么用。
楼主还是看看文件是不是被其他程序占用着，或者你代码有什么问题吧。 CStdioFile f1;
f1.Open(_T("F:\\应用软件安装包\\xxx.rar"),
CFile::modeRead | CFile::typeBinary);
f1.Seek(1024*1024*512, CFile::begin);
char TextBuffer[512] = {0};
f1.Read(TextBuffer, 512);其中"F:\\应用软件安装包\\xxx.rar"这个文件2GB，f1.Read(TextBuffer, 512);立马完成，楼主也可用此代码测试下你那个1.5GB或其他文件。

貌似不是Seek的问题。
用CFile代替CStdioFile试试。

赞同这个说法，话说文件系统支持的文件长度是ULONGLONG数据类型的，那是多大，2G算的上什么的？LZ是否使用了多线程，造成了访问互锁？

赞同这个说法，话说文件系统支持的文件长度是ULONGLONG数据类型的，那是多大，2G算的上什么的？LZ是否使用了多线程，造成了访问互锁？嗯啊，是多线程，但是只有个线程访问这个文件。

赞同这个说法，话说文件系统支持的文件长度是ULONGLONG数据类型的，那是多大，2G算的上什么的？LZ是否使用了多线程，造成了访问互锁？嗯啊，是多线程，但是只有个线程访问这个文件。弄个测试工程，就一个线程，试试你的文件读写，我感觉是你多线程发生长时间互锁

SetFilePointer
The SetFilePointer function moves the file pointer of an open file. DWORD SetFilePointer(
  HANDLE hFile,          // handle of file
  LONG lDistanceToMove,  // number of bytes to move file pointer
  PLONG lpDistanceToMoveHigh,
                         // pointer to high-order DWORD of
                         // distance to move
  DWORD dwMoveMethod     // how to move
);

Parameters
hFile
Handle to the file whose file pointer is to be moved. The file handle must have been created with GENERIC_READ or GENERIC_WRITE access to the file.
lDistanceToMove
Low-order 32 bits of a signed value that specifies the number of bytes to move the file pointer. If lpDistanceToMoveHigh is not NULL, lpDistanceToMoveHigh and lDistanceToMove form a single 64-bit signed value that specifies the distance to move. If lpDistanceToMoveHigh is NULL, lDistanceToMove is a 32-bit signed value. A positive value for lDistanceToMove moves the file pointer forward in the file, and a negative value moves the file pointer backward.
lpDistanceToMoveHigh
Pointer to the high-order 32 bits of the signed 64-bit distance to move. If you do not need the high-order 32 bits, this pointer may be NULL. When non-NULL, this parameter also receives the high-order DWORD of the new value of the file pointer. For more information, see the Res section later in this topic.
dwMoveMethod
Starting point for the file pointer move. This parameter can be one of the following values. Value Meaning
FILE_BEGIN The starting point is zero or the beginning of the file.
FILE_CURRENT The starting point is the current value of the file pointer.
FILE_END The starting point is the current end-of-file position.
Return Values
If the SetFilePointer function succeeds and lpDistanceToMoveHigh is NULL, the return value is the low-order DWORD of the new file pointer. If lpDistanceToMoveHigh is not NULL, the function returns the low order DWORD of the new file pointer, and puts the high-order DWORD of the new file pointer into the LONG pointed to by that parameter. If the function fails and lpDistanceToMoveHigh is NULL, the return value is 0xFFFFFFFF. To get extended error information, call GetLastError. If the function fails, and lpDistanceToMoveHigh is non-NULL, the return value is 0xFFFFFFFF. However, because 0xFFFFFFFF is a valid value for the low-order DWORD of the new file pointer, you must check GetLastError to determine whether an error occurred. If an error occurred, GetLastError returns a value other than NO_ERROR. For a code example that illustrates this point, see the Res section later in this topic.If the new file pointer would have been a negative value, the function fails, the file pointer is not moved, and the code returned by GetLastError is ERROR_NEGATIVE_SEEK.Res
You cannot use the SetFilePointer function with a handle to a nonseeking device, such as a pipe or a communications device. To determine the file type for hFile, use the GetFileType function.To determine the present position of a file pointer, see Retrieving a File Pointer.You should be careful when setting the file pointer in a multithreaded application. For example, an application whose threads share a file handle, update the file pointer, and read from the file must protect this sequence by using a critical section object or mutex object. For more information about these objects, see Critical Section Objects and Mutex Objects.If the hFile file handle was opened with the FILE_FLAG_NO_BUFFERING flag set, an application can move the file pointer only to sector-aligned positions. A sector-aligned position is a position that is a whole number multiple of the volume's sector size. An application can obtain a volume's sector size by calling the GetDiskFreeSpace function. If an application calls SetFilePointer with distance-to-move values that result in a position that is not sector-aligned and a handle that was opened with FILE_FLAG_NO_BUFFERING, the function fails, and GetLastError returns ERROR_INVALID_PARAMETER.Note that it is not an error to set the file pointer to a position beyond the end of the file. The size of the file does not increase until you call the SetEndOfFile, WriteFile, or WriteFileEx function. A write operation increases the size of the file to the file pointer position plus the size of the buffer written, leaving the intervening bytes uninitialized. If the return value is 0xFFFFFFFF and if lpDistanceToMoveHigh is non-NULL, an application must call GetLastError to determine whether the function has succeeded or failed. The following sample code illustrates this point: //
// Case One: calling the function with lpDistanceToMoveHigh == NULL

// Try to move hFile's file pointer some distance
dwPtr = SetFilePointer (hFile, lDistance, NULL, FILE_BEGIN) ;

if (dwPtr == 0xFFFFFFFF) // Test for failure
{
    // Obtain the error code
    dwError = GetLastError() ;

    // Deal with failure
    // . . .

} // End of error handler
//
// Case Two: calling the function with lpDistanceToMoveHigh != NULL

// Try to move hFile's file pointer some huge distance
dwPtrLow = SetFilePointer (hFile, lDistLow, & lDistHigh, FILE_BEGIN) ;

// Test for failure
if (dwPtrLow == 0xFFFFFFFF && (dwError = GetLastError()) != NO_ERROR )
{
    // Deal with failure
    // . . . } // End of error handler

The parameter lpDistanceToMoveHigh is used to manipulate huge files. If it is set to NULL, then lDistanceToMove has a maximum value of 2^31–2, or 2 gigabytes less two. This is because all file pointer values are signed values. Therefore if there is even a small change that the file will grow to that size, you should treat the file as a huge file and work with 64-bit file pointers. With file compression on NTFS, and sparse files, it is possible to have files that large even if the underlying volume is not that large.If lpDistanceToMoveHigh is not NULL, then lpDistanceToMoveHigh and lDistanceToMove form a single 64-bit signed value. The lDistanceToMove parameter is treated as the low-order 32 bits of the value, and lpDistanceToMoveHigh as the upper 32 bits. Thus, lpDistanceToMoveHigh is a sign extension of lDistanceToMove.To move the file pointer from zero to 2 gigabytes, lpDistanceToMoveHigh can be either NULL or a sign extension of lDistanceToMove. To move the pointer more than 2 gigabytes, use lpDistanceToMoveHigh and lDistanceToMove as a single 64-bit quantity. For example, to move in the range from 2 gigabytes to 4 gigabytes set the contents of lpDistanceToMoveHigh to zero, or to –1 for a negative sign extension of lDistanceToMove.To work with 64-bit file pointers, you can declare a LONG, treat it as the upper half of the 64-bit file pointer, and pass its address in lpDistanceToMoveHigh. This means you have to treat two different variables as a logical unit, which is error-prone. The problems can be ameliorated by using the LARGE_INTEGER structure to create a 64-bit value and passing the two 32-bit values by means of the appropriate elements of the union.It is conceptually simpler and better design to use a function to hide the interface to SetFilePointer. To do so, use something like this:__int64 myFileSeek (HANDLE hf, __int64 distance, DWORD MoveMethod)
{
   LARGE_INTEGER li;   li.QuadPart = distance;   li.LowPart = SetFilePointer (hf, li.LowPart, &li.HighPart, MoveMethod);   if (li.LowPart == 0xFFFFFFFF && GetLastError() != NO_ERROR)
   {
      li.QuadPart = -1;
   }   return li.QuadPart;
}Note  You can use SetFilePointer to determine the length of a file. To do this, use FILE_END for dwMoveMethod and seek to location zero. The file offset returned is the length of the file. However, this practice can have unintended side effects, such as failure to save the current file pointer so that the program can return to that location. It is simpler and safer to use GetFileSize instead.You can also use the SetFilePointer function to query the current file pointer position. To do this, specify a move method of FILE_CURRENT and a distance of zero. QuickInfo
  Windows NT: Requires version 3.1 or later.
  Windows: Requires Windows 95 or later.
  Windows CE: Requires version 1.0 or later.
  Header: Declared in winbase.h.
  Import Library: Use kernel32.lib.See Also
File I/O Overview, File Functions, GetDiskFreeSpace, GetFileSize, GetFileType, ReadFile, ReadFileEx, ReadFileVlm, WriteFile, WriteFileEx, WriteFileVlm
CStdioFile由CFile派生CFile Class Members
Data Membersm_hFile Usually contains the operating-system file handle.

推荐使用WinHex软件查看硬盘或文件或内存中的原始字节内容。

调试易

大文件读取问题 C++/MFC

解决方案 »