当前位置:网站首页>Implementation of inserting millions of data into MySQL database in 10 seconds
Implementation of inserting millions of data into MySQL database in 10 seconds
2022-04-23 11:06:00 【liming89】
Source of the article : Learn through http://www.bdgxy.com/
First of all, let's think about a question :
To insert such a huge amount of data into the database , Under normal circumstances, there must be frequent visits , I can't afford any kind of machinery and equipment . So how to avoid frequent access to the database , Can you make a visit , And then execute it ?
Java In fact, it has given us the answer .
Here we need to use two key objects :Statement、PrepareStatement
Let's take a look at the characteristics of both :

What to use BaseDao Tool class (jar package / Maven rely on ) (Maven The dependent code is attached at the end of the text )( Packaged for ease of use )

notes :( a key )rewriteBatchedStatements=true, Insert multiple data at a time , Just insert it once !!
public class BaseDao { // Static tool class , Used to create database connection objects and release resources , Convenient to call // Import driver jar Package or add Maven rely on ( What we use here is Maven,Maven The dependent code is attached at the end of the text ) static { try { Class.forName("com.mysql.cj.jdbc.Driver"); } catch (ClassNotFoundException e) { e.printStackTrace(); } }// Get database connection object public static Connection getConn() { Connection conn = null; try { // rewriteBatchedStatements=true, Insert multiple data at a time , Just insert it once conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/million-test?rewriteBatchedStatements=true", "root", "qwerdf"); } catch (SQLException throwables) { throwables.printStackTrace(); } return conn; } // Release resources public static void closeAll(AutoCloseable... autoCloseables) { for (AutoCloseable autoCloseable : autoCloseables) { if (autoCloseable != null) { try { autoCloseable.close(); } catch (Exception e) { // TODO Auto-generated catch block e.printStackTrace(); } } } }
}
Next, the key codes and comments :
/* Because the processing speed of the database is amazing The single throughput is very large Very efficient execution
addBatch() Put a number of sql Statements are loaded together , Then it is sent to the database for execution at one time , Execution takes a short time
and preparedStatement.executeUpdate() It is sent to the database one by one for execution Time is spent on the transmission of database connection */
public static void main(String[] args) {
long start = System.currentTimeMillis(); // Get the current time of the system , Record before method execution
Connection conn = BaseDao.getConn(); // Call the static tool class just written to get the connection database object
String sql = "insert into mymilliontest values(null,?,?,?,NOW())"; // Executes sql sentence
PreparedStatement ps = null;
try {
ps = conn.prepareStatement(sql); // obtain PreparedStatement object
// Continuous production sql
for (int i = 0; i < 1000000; i++) {
ps.setString(1, Math.ceil(Math.random() * 1000000) + "");
ps.setString(2, Math.ceil(Math.random() * 1000000) + "");
ps.setString(3, UUID.randomUUID().toString()); // UUID This class is used to randomly generate a string that will not be repeated
ps.addBatch(); // Adds a set of parameters to this PreparedStatement Object in the batch command .
}
int[] ints = ps.executeBatch();// Submit a batch of commands to the database for execution , If all commands are executed successfully , Returns an array of update counts .
// If the array length is not 0, shows sql Statement executed successfully , That is, one million pieces of data are added successfully !
if (ints.length > 0) {
System.out.println(" One million pieces of data have been successfully added !!");
}
} catch (SQLException throwables) {
throwables.printStackTrace();
} finally {
BaseDao.closeAll(conn, ps); // Call the static tool class just written to release resources
}
long end = System.currentTimeMillis(); // Get the system time again
System.out.println(" Time used :" + (end - start) / 1000 + " second "); // Subtracting the two times is the time taken for the method to execute
}
Finally, let's run to see the effect :


Hey , It's longer than 10 second , Equipment almost means , Hope to understand, huh ~

<!-- Used to connect to the database mysql-connector-java rely on -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.27</version>
</dependency>
PS : Adding threads will be faster , Examples will be provided in subsequent articles .
This is about MySQL database 10 This is the end of the article on the implementation of inserting millions of data in seconds , More about MySQL Insert millions of data Please search rookie tutorial www.piaodoo.com Previous articles or continue to browse the relevant articles below. I hope you can support rookie tutorials in the future www.piaodoo.com!
版权声明
本文为[liming89]所创,转载请带上原文链接,感谢
https://yzsam.com/2022/04/202204231102124393.html
边栏推荐
- Jupyter Lab 十大高生产力插件
- 全栈交叉编译X86完成过程经验分享
- CUMCM 2021-B:乙醇偶合制备C4烯烃(2)
- 26. Delete duplicates in ordered array
- Learning note 5 - gradient explosion and gradient disappearance (k-fold cross verification)
- Typora operation skill description (I)
- Qinglong panel pull library command update [April 20, 2022] collection is not lost
- CUMCM 2021-B:乙醇偶合制備C4烯烴(2)
- Visualized common drawing (II) line chart
- Upgrade the functions available for cpolar intranet penetration
猜你喜欢

关于JUC三大常用辅助类

The courses bought at a high price are open! PHPer data sharing

升级cpolar内网穿透能获得的功能

26. Delete duplicates in ordered array

Excel·VBA自定义函数获取单元格多数值

Database management software sqlpro for SQLite for Mac 2022.30

Structure of C language (Advanced)

Notes on concurrent programming of vegetables (IX) asynchronous IO to realize concurrent crawler acceleration

Excel·VBA数组冒泡排序函数

About the three commonly used auxiliary classes of JUC
随机推荐
Mba-day6 logic - hypothetical reasoning exercises
如何使用JDBC CallableStatement.wasNull()方法调用来查看最后一个OUT参数的值是否为 SQL NULL
Cygwin 中的 rename 用法
Mysql系列SQL查询语句书写顺序及执行顺序详解
MySQL Router重装后重新连接集群进行引导出现的——此主机中之前已配置过的问题
Visual common drawing (I) stacking diagram
MySQL数据库10秒内插入百万条数据的实现
关于JUC三大常用辅助类
Visualization Road (11) detailed explanation of Matplotlib color
Cumcm 2021 - B: préparation d'oléfines C4 par couplage éthanol (2)
Google Earth Engine(GEE)——将原始影像进行升尺度计算(以海南市为例)
Mba-day5 Mathematics - application problems - engineering problems
How to bind a process to a specified CPU
The difference between restful and soap
Notes on concurrent programming of vegetables (IX) asynchronous IO to realize concurrent crawler acceleration
VIM usage
Strongest date regular expression
Facing the global market, platefarm today logs in to four major global platforms such as Huobi
Detailed introduction to paging exploration of MySQL index optimization
软件测试人员,如何优秀的提Bug?