painfully slow string concatenation

Hi,
Trying to get faster (text) file reading time.
I was reading a series of text files, and concatenating their contents into
a large string.
this is unbelievably slow.

if i just read them, 1096 files are read in less than a second so it's not
the reading that's slow as I originally thought.

putting the strings into a collection is also fast (seemingly immediate -
need to get a timer working to check actual)

is concatenating inherently slow? probably because internally a string
array is being created and redimmed repeatedly?

just curious

 

Why are you concatenating strings, anyway?

 

String concatenation is slow. Here is an article on it and how to improve
it. http://www.vb2themax.com/HtmlDoc.asp?Table=Articles&ID=30

I also posted a CFileCopy class by Karl Peterson in news group
autodesk.autocad.customer-files on 6/3/2003. Subject line was "PLT files".
It is a class I use to concatenate hundrededs of plot files into one large
plot job. For some reason Google doesn't show it in the search results.

LJB

 

Out of curiousity, what about sizing your string one time and then loading
the substrings into it using Mid() function. Does this example scale up
well to your problem? I don't know if Mid() is a slow function or not.

James

Sub test_Concat()

Dim a As String 'concatenated string
Dim b(0 To 7) As String 'substrings
Dim aLen As Long, curByte As Long 'length of final string
Dim strings As New Collection

b(0) = "abcde"
b(1) = "fghijklmn"
b(2) = ""
b(3) = "opq"
b(4) = "rstuv"
b(5) = "wxyz"
b(6) = "0123456"
b(7) = "789"

For i = LBound(b) To UBound(b)
strings.Add b(i)
Next 'i

aLen = 0
For Each curstr In strings
aLen = aLen + Len(curstr)
Next 'curStr

a = Space(aLen)

curByte = 1
'I'm always nervous trusting For...Each to traverse the collection in the
proper order, but
' I guess it does, because the alphabet is not scrambled.
For Each curstr In strings
Mid(a, curByte, Len(curstr)) = curstr
curByte = curByte + Len(curstr)
Next 'curStr
Debug.Print "after: '" & a & "'"

End Sub

 

As Tony asked: why are you doing this?

Rather than read one line at a time and concatenating the results, you
might try opening the file in Binary mode and reading a number of
characters equal to the file length. If you prefer, you can use the
ReadAll method of the TextStream class. You'll find the TextStream
class in the Scripting Runtime library.

 

Thanks Frank,
I'll try those also.

see my reply to him - just my usual idiocy!

 

You got it. Every time you concatenate a string a new one is created,
the contents of both are copied into the new one, and any cleanup
necessary takes place. Not bad for small strings, but with many large
strings it becomes very slow. There are some options though...

1. Load each string as one element of an array and use join to
combine them.

2. Allocate a big string yourself and place each substring into it
yourslef. ie:
s = space(10000)
mid$(s,10,100) = tempstring ' Place tempstring at position 10
mid$(s,110,200)=tempstring2

-Tim

 

thanks for the link, i'm checking it out

 

Wouldn't Windows Find/Search work for what you need?

 

it absolutely does work for what I need.
I was trying to beat it's time in searching, and also be able to do it
inside acad instead of opening or tabbing to explorer, browsing to the
folder, filling out the fields, etc.
besides, what's the fun of letting someone else do your work for you???
I guess to get the speed I'm thinking about I'd have to learn to write
assembler? not in this lifetime!

now that I see that vb is actually reading the files faster than lisp is, if
I can rewrite all my lisp code that finds the functions into vb code, then I
can test a vb approach.

Except I have to use lisp to call vb and that always is really slow to call
up a dvb from lisp, don't understand what the delay is but it's always been
noticable waiting for a vba dvb to load and start. not sure if calling a
dll will be faster,

It may still probably be faster that starting explorer, browsing, entering
fields etc.
we'll see....