Click here to Skip to main content
15,844,152 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
There are 60 or more linux client machines, each having a dedicated directory containing 100 or more data files. Each file is of size 1GB. All these files are in dedicated paths and must be transferred to server everyday 12AM.

I need a solution which can transfer these data file to server automatically every day at 12AM.

I need a idea how to resolve the problem. Please guide me how can I do this? 


What I have tried:

I wrote a multithreaded client ser ver application where the serve accepts Multiple clients and transfer a simple file from client to server.
Posted
Updated 15-Jul-22 11:48am
Comments
Greg Utas 14-Jul-22 16:40pm    
So how did your multithreaded server work out? That's how I would also do it, so I'm not sure what you're asking.
Dave Kreskowiak 14-Jul-22 18:04pm    
Why would a CRON job not handle this? It would launch a series of copy commands to copy the files to a network share on your server.

Not a single line of code required.
k5054 14-Jul-22 18:29pm    
Lets take a quick look at your requirements:
60 + clients
100 + files @ 1 GB each
I make that over 6 TB of data to transfer. With a 1 GB NIC, That's going to take about 90 minutes to transfer, assuming there's nothing else using bandwidth at the same time.
First question: Can the data be compressed for transfer?
Second question: Would some sort of network filesystem (e.g NFS or Samba) be an acceptable solution? That would mean that rather than uploading the files to the server, the files would always be available to the server. So, rather than updating the server once per day, updates would be live, as and when they occur on the client.
Gerry Schmitz 14-Jul-22 19:24pm    
Hard to picture any client generating 100 GB of data per day. That's like 4 days of HD video ... per day.
steveb 16-Jul-22 12:22pm    
Stock bid ask recorded data easily goes to 1 TB per day file just for one security

1 solution

You do not need multithreaded socket programming for that. All you need to do is to map the network drive of the remote computer and copy files over with a batch command. And if you want to automate this use operating system's scheduler that will execute the batch file at the specified time
 
Share this answer
 
v2

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900