用ksh脚本从oracle数据库中导出80w数据到csv文件,如用户给定名字为a.csv(文件最大4000行记录),则自动生产文件为a_1.csv,a_2.csv,...., a_200.csv我已经实现了一个方法,但80w要导5小时,用户没法接受。如下:
sqlplus -s user/pwd @${SqlDir}/select_log.sql >/tmp/a.tmp
然后每read 4000行到文件a_1.csv,并且每隔4000行换一个文件名。大家有什么好方法吗?注意效率
sqlplus -s user/pwd @${SqlDir}/select_log.sql >/tmp/a.tmp
然后每read 4000行到文件a_1.csv,并且每隔4000行换一个文件名。大家有什么好方法吗?注意效率
再用split分割文件
set pages 0
set head off
set echo off
set feedback off
spool spool_text.csv
select col1, col2 from my_table where rownum<1000000;
spool off
exit## shell script to spool out:
sqlplus -s user/password @spool_test.sql > /dev/null## split file:
split -l 4000 spool_text.csv splited
echo "SET HEADING OFF;">${SqlDir}/select_log.sql
echo "SET FEEDBACK OFF;">>${SqlDir}/select_log.sql
echo "SET LINESIZE 10000;">>${SqlDir}/select_log.sql
echo "SPOOL ${SqlDir}/EXPORT_ALL.CSV;" >> ${SqlDir}/select_log.sql
echo "SELECT SALESMAN_ID||','||">>${SqlDir}/select_log.sql
echo "SALESMAN_NAME||','||">>${SqlDir}/select_log.sql
echo "SALES_AMOUNT||','||">>${SqlDir}/select_log.sql
echo "SALES_DATE from sales_range;">>${SqlDir}/select_log.sql
echo "SPOOL OFF;" >> ${SqlDir}/select_log.sql
echo exit | ${ORACLE_HOME}/bin/sqlplus -s omc_sbardz812/omc_sbardz812 @${SqlDir}
/select_log.sql >/dev/nullsplit -l 4000 -a 5 ${SqlDir}/EXPORT_ALL.CSV log_20060606_
=============================================================以上是我的脚本,80w数据我用了3个多小时还没完呢。有问题吗?
另外,很重要的一点,split后,再不是csv后缀了。看来还得文件遍历,然后一个一个的mv 成csv文件。
再利用脚本导出来,看看速度应该有所提高。不过有一点必须明白,全扫描8w的表数据肯定不是几分钟可以完成的,何况要写成物理文件。
800,000 records
size of each record (line) = 9GB/800,000 = 12,079 Bytesselect 中只有 4 列(SALESMAN_ID, SALESMAN_NAME, SALES_AMOUNT, SALES_DATE
),你觉得每行纪录占12KB合理吗?
解决这个问题很简单,在spool开始之前加上两行set命令set trimspool on
set termout off第一个set是指定trim spool出的每行在行末的空格
第二个set是指定只spool而不回显,所以在运行sqlplus的时候不需要 >/dev/null。
set pagesize 0
before spool to suppress all headings, page breaks, titles, the initial blank line, and other formatting information.
set pagesize 0
before spool to suppress all headings, page breaks, titles, the initial blank line, and other formatting information.
echo "SET FEEDBACK OFF;">>${SqlDir}/select_log.sql
echo "SET LINESIZE 500;">>${SqlDir}/select_log.sql
echo "SET trimspool on;">>${SqlDir}/select_log.sql
echo "SET termout off;" >>${SqlDir}/select_log.sql
echo "SET NEWPAGE NONE;" >>${SqlDir}/select_log.sql
echo "SPOOL ${SqlDir}/EXPORT_ALL.CSV;" >> ${SqlDir}/select_log.sql
echo "SELECT SALESMAN_ID||','||">>${SqlDir}/select_log.sql
echo "SALESMAN_NAME||','||">>${SqlDir}/select_log.sql
echo "SALES_AMOUNT||','||">>${SqlDir}/select_log.sql
echo "SALES_DATE from sales_range;">>${SqlDir}/select_log.sql
echo "SPOOL OFF;" >> ${SqlDir}/select_log.sql
echo exit | ${ORACLE_HOME}/bin/sqlplus -s user/pwd @${SqlDir}/select_log.sql split -l 4000 -a 3 ${SqlDir}/EXPORT_ALL.CSV log_20060606_ #mv these splitted file to be csv file
for file in $(ls ./)
do
echo $file | grep log_20060606_ > /dev/null
status=$?
if [ $status -eq 0 ]; then
mv $file $file.csv
fi
status=1
done 80万10分钟就可以搞定了。ksh可以执行,由于有这句:$(ls ./),sh不支持。