Partial data insertion in postgresql database using spring boot service

Question

1.00/5 (1 vote)

See more:

I'm inserting large volume of data into PostgreSQL database using spring boot approximate about 1 million records, these data comes in batches during API calls and I'm saving each batch as the whole in database but the only challenge facing right now is that only half of data is inserted into database about 500k records instead of whole 1 million records I don't know where I'm doing it wrong can you please provide some suggestions, these batches comes from the desktop application already divided into groups of 50000 records here below are my source codes for both spring boot application and desktop application

What I have tried:

Service class 

@Service 
public class OrderProcessingService {

@Autowired
private OrderRepository orderRepository;

@Async
public void process_data(List<Order> order_list) {
          
visitRepository.saveAll(order_list);
    }   

}

Rest Api

@RestController
@RequestMapping("/api")
public class DataUploading {
    
 @Autowired
 private OrderProcessingService order_processing_service;
    
@PostMapping("/orders/{hfrcode}")
 private void get_orders(@RequestBody List<Order> order) {
 order_processing_service.process_data(order);
}

}

Desktop App data submission service

 int batchSize = 40000;
 for(int i=0; i< order_list.Count; i+= batchSize) {
 var batch = order_list.Skip(i).Take(batchSize);
 httpClient.PostAsJsonAsync("api/orders/" + hfrcode, batch);
 }

Posted 21-Mar-24 5:05am

Office Systems

Add a Solution

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Andre Oosthuizen · Accepted Answer · 2024-03-21T21:42:00

Your issue is in the way you are invoking the async 'process_data' method in the 'OrderProcessingService' class. The method is being executed in a separate thread, however, the problem is that your 'process_data' method is not returning anything, and it's not being awaited, which means your program does not wait for the method to finish before proceeding to the next iteration of the loop. As a result, some of your batches might not be saved completely before the loop continues, leading to data loss - Creating Asynchronous Methods [^]

Your code should execute using something similar to this -

In your Service Class(OrderProcessingService) -

Java

import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.util.List;

@Service
public class OrderProcessingService {

    private final OrderRepository orderRepository;

    public OrderProcessingService(OrderRepository orderRepository) {
        this.orderRepository = orderRepository;
    }

    @Transactional
    public void processBatch(List<Order> orderList) {
        orderRepository.saveAll(orderList);
    }
}

In your Rest API Class(DataUploading) -

Java

import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.List;

@RestController
@RequestMapping("/api")
public class DataUploading {

    private final OrderProcessingService orderProcessingService;

    public DataUploading(OrderProcessingService orderProcessingService) {
        this.orderProcessingService = orderProcessingService;
    }

    @PostMapping("/orders/{hfrcode}")
    public void getOrders(@RequestBody List<Order> orders) {
        orderProcessingService.processBatch(orders);
    }
}

Lastly, in your Desktop App Data Submission Service -

Java

int batchSize = 40000;
for (int i = 0; i < orderList.size(); i += batchSize) {
    List<Order> batch = orderList.subList(i, Math.min(i + batchSize, orderList.size()));
    orderProcessingService.processBatch(batch);
}

In the '@Transactional' annotation in the 'processBatch' above, the method ensures that the entire batch is saved atomically, rolling back the transaction if any part of the batch fails to save.