Click here to Skip to main content
14,546,974 members
   

Database

 
AnswerRe: Replace ids with value Pin
Eddy Vluggen16-Mar-18 8:54
professionalEddy Vluggen16-Mar-18 8:54 
Rant[REPOST] Replace ids with value Pin
Richard Deeming16-Mar-18 11:42
mveRichard Deeming16-Mar-18 11:42 
QuestionReplace Ids With value Pin
Member 1266749416-Mar-18 8:29
MemberMember 1266749416-Mar-18 8:29 
AnswerRe: Replace Ids With value Pin
Victor Nijegorodov16-Mar-18 9:21
MemberVictor Nijegorodov16-Mar-18 9:21 
QuestionDB2 to SQL conversion Pin
Member 1028121715-Mar-18 4:06
MemberMember 1028121715-Mar-18 4:06 
AnswerRe: DB2 to SQL conversion Pin
Victor Nijegorodov15-Mar-18 5:09
MemberVictor Nijegorodov15-Mar-18 5:09 
AnswerRe: DB2 to SQL conversion Pin
Richard Deeming15-Mar-18 6:41
mveRichard Deeming15-Mar-18 6:41 
QuestionBillion rows Pin
jschell10-Mar-18 10:39
Memberjschell10-Mar-18 10:39 
Just a random question since I already have a different solution but I had the following situation.

I have a table with a billion rows (actually probably about 1.1 billion.)

I was trying to span the table, read every row and do an analysis.

Certainly couldn't load the entire table. I was using a paged query (limit/count). Each page took about 90 minutes for the query itself. So not really something that was going to allow me to do much analysis.

Any other ideas on spanning it or speeding it up?

At one point I was even considering just dumping it and writing an app to do the analysis outside of the database. That was about the only other solution I had.

The database was MySQL (AWS Aurora actually).

The relevant parts of the table were as follows and the id has a primary key. (I didn't design the table.)

Table T
     id char(22)
     RefId1 char(22)
     RefId2 char(22)
     ...

AnswerRe: Billion rows Pin
Jörgen Andersson10-Mar-18 11:42
professionalJörgen Andersson10-Mar-18 11:42 
GeneralRe: Billion rows Pin
jschell14-Mar-18 15:13
Memberjschell14-Mar-18 15:13 
GeneralRe: Billion rows Pin
Mycroft Holmes14-Mar-18 17:29
professionalMycroft Holmes14-Mar-18 17:29 
GeneralRe: Billion rows Pin
jschell17-Mar-18 5:58
Memberjschell17-Mar-18 5:58 
GeneralRe: Billion rows Pin
Mycroft Holmes17-Mar-18 13:18
professionalMycroft Holmes17-Mar-18 13:18 
GeneralRe: Billion rows Pin
jschell24-Mar-18 6:14
Memberjschell24-Mar-18 6:14 
GeneralRe: Billion rows Pin
Mycroft Holmes24-Mar-18 13:08
professionalMycroft Holmes24-Mar-18 13:08 
GeneralRe: Billion rows Pin
jschell31-Mar-18 7:01
Memberjschell31-Mar-18 7:01 
GeneralRe: Billion rows Pin
Mycroft Holmes31-Mar-18 13:40
professionalMycroft Holmes31-Mar-18 13:40 
GeneralRe: Billion rows Pin
Jörgen Andersson31-Mar-18 23:13
professionalJörgen Andersson31-Mar-18 23:13 
GeneralRe: Billion rows Pin
jschell6-Apr-18 10:03
Memberjschell6-Apr-18 10:03 
AnswerRe: Billion rows Pin
Jörgen Andersson11-Mar-18 22:56
professionalJörgen Andersson11-Mar-18 22:56 
GeneralRe: Billion rows Pin
jschell14-Mar-18 15:15
Memberjschell14-Mar-18 15:15 
GeneralRe: Billion rows Pin
Jörgen Andersson14-Mar-18 22:17
professionalJörgen Andersson14-Mar-18 22:17 
GeneralRe: Billion rows Pin
jschell17-Mar-18 5:52
Memberjschell17-Mar-18 5:52 
GeneralRe: Billion rows Pin
Jörgen Andersson17-Mar-18 7:39
professionalJörgen Andersson17-Mar-18 7:39 
GeneralRe: Billion rows Pin
jschell24-Mar-18 6:18
Memberjschell24-Mar-18 6:18 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.