Why is it so slow when assigning a concatenated string to a variable in python?How to concatenate text from multiple rows into a single text string in SQL server?How do I parse a string to a float or int in Python?Python join: why is it string.join(list) instead of list.join(string)?How to substring a string in Python?Reverse a string in PythonConverting integer to string in Python?How do I concatenate two lists in Python?Does Python have a string 'contains' substring method?How to concatenate string variables in BashHow do I lowercase a string in Python?
How to pronounce "r" after a "g"?
Why was the Ancient One so hesitant to teach Dr. Strange the art of sorcery?
Why is it so slow when assigning a concatenated string to a variable in python?
Is it a Munchausen Number?
"Right on the tip of my tongue" meaning?
What does i386 mean on macOS Mojave?
Why does a C.D.F need to be right-continuous?
How can a Lich look like a human without magic?
What is the best way for a skeleton to impersonate human without using magic?
Control variables and other independent variables
What are the ramifications of setting ARITHABORT ON for all connections in SQL Server?
What are some possible reasons that a father's name is missing from a birth certificate - England?
Why do Thanos's punches not kill Captain America or at least cause some mortal injuries?
Why do unstable nuclei form?
Is there any evidence to support the claim that the United States was "suckered into WW1" by Zionists, made by Benjamin Freedman in his 1961 speech?
How did Thanos not realise this had happened at the end of Endgame?
Why did God specifically target the firstborn in the 10th plague (Exodus 12:29-36)?
Is Simic Ascendancy triggered by Awakening of Vitu-Ghazi?
Pre-1993 comic in which Wolverine's claws were turned to rubber?
Noob at soldering, can anyone explain why my circuit won't work?
How to select certain lines (n, n+4, n+8, n+12...) from the file?
Drawing Quarter-Circle
Is there enough time to Planar Bind a creature conjured by a 1-hour-duration spell?
Was there ever any real use for a 6800-based Apple I?
Why is it so slow when assigning a concatenated string to a variable in python?
How to concatenate text from multiple rows into a single text string in SQL server?How do I parse a string to a float or int in Python?Python join: why is it string.join(list) instead of list.join(string)?How to substring a string in Python?Reverse a string in PythonConverting integer to string in Python?How do I concatenate two lists in Python?Does Python have a string 'contains' substring method?How to concatenate string variables in BashHow do I lowercase a string in Python?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
If it is only concatenation of strings as follows, it finish immediately.
test_str = "abcdefghijklmn123456789"
str1 = ""
str2 = ""
start = time.time()
for i in range(1, 100001):
str1 = str1 + test_str
str2 = str2 + test_str
if i % 20000 == 0:
print("time(sec) => ".format(time.time() - start))
start = time.time()
Constant processing time
time(sec) => 0.013324975967407227
time(sec) => 0.020363807678222656
time(sec) => 0.009979963302612305
time(sec) => 0.01744699478149414
time(sec) => 0.0227658748626709
Inexplicably, assigning a concatenated string to another variable makes the process slower and slower.
test_str = "abcdefghijklmn123456789"
str1 = ""
str2 = ""
start = time.time()
for i in range(1, 100001):
str1 = str1 + test_str
# str2 = str2 + test_str
# ↓
str2 = str1
if i % 20000 == 0:
print("time(sec) => ".format(time.time() - start))
start = time.time()
Processing time will be delayed.
time(sec) => 0.36466407775878906
time(sec) => 1.105351209640503
time(sec) => 2.6467738151550293
time(sec) => 5.891657829284668
time(sec) => 9.266698360443115
Both python2 and python3 give the same result.
why?????
python string-concatenation
New contributor
add a comment |
If it is only concatenation of strings as follows, it finish immediately.
test_str = "abcdefghijklmn123456789"
str1 = ""
str2 = ""
start = time.time()
for i in range(1, 100001):
str1 = str1 + test_str
str2 = str2 + test_str
if i % 20000 == 0:
print("time(sec) => ".format(time.time() - start))
start = time.time()
Constant processing time
time(sec) => 0.013324975967407227
time(sec) => 0.020363807678222656
time(sec) => 0.009979963302612305
time(sec) => 0.01744699478149414
time(sec) => 0.0227658748626709
Inexplicably, assigning a concatenated string to another variable makes the process slower and slower.
test_str = "abcdefghijklmn123456789"
str1 = ""
str2 = ""
start = time.time()
for i in range(1, 100001):
str1 = str1 + test_str
# str2 = str2 + test_str
# ↓
str2 = str1
if i % 20000 == 0:
print("time(sec) => ".format(time.time() - start))
start = time.time()
Processing time will be delayed.
time(sec) => 0.36466407775878906
time(sec) => 1.105351209640503
time(sec) => 2.6467738151550293
time(sec) => 5.891657829284668
time(sec) => 9.266698360443115
Both python2 and python3 give the same result.
why?????
python string-concatenation
New contributor
3
Because you are using a quadratic time algorithm. The interpreter is able to optimize this, but only in some cases. You should not rely on that, and instead, use a linear time algorithm (generally, append to a list then''.join
)
– juanpa.arrivillaga
2 hours ago
As a result of verifying it according to @shadowranger answer, The cause was found. If only string concatenation is used, the ids of str1 and str2 remain the same, but if str1 is assigned to str2, the id of str1 changes each time. In other words, it is slowed by str1 being allocated every time.
– uma66
1 hour ago
add a comment |
If it is only concatenation of strings as follows, it finish immediately.
test_str = "abcdefghijklmn123456789"
str1 = ""
str2 = ""
start = time.time()
for i in range(1, 100001):
str1 = str1 + test_str
str2 = str2 + test_str
if i % 20000 == 0:
print("time(sec) => ".format(time.time() - start))
start = time.time()
Constant processing time
time(sec) => 0.013324975967407227
time(sec) => 0.020363807678222656
time(sec) => 0.009979963302612305
time(sec) => 0.01744699478149414
time(sec) => 0.0227658748626709
Inexplicably, assigning a concatenated string to another variable makes the process slower and slower.
test_str = "abcdefghijklmn123456789"
str1 = ""
str2 = ""
start = time.time()
for i in range(1, 100001):
str1 = str1 + test_str
# str2 = str2 + test_str
# ↓
str2 = str1
if i % 20000 == 0:
print("time(sec) => ".format(time.time() - start))
start = time.time()
Processing time will be delayed.
time(sec) => 0.36466407775878906
time(sec) => 1.105351209640503
time(sec) => 2.6467738151550293
time(sec) => 5.891657829284668
time(sec) => 9.266698360443115
Both python2 and python3 give the same result.
why?????
python string-concatenation
New contributor
If it is only concatenation of strings as follows, it finish immediately.
test_str = "abcdefghijklmn123456789"
str1 = ""
str2 = ""
start = time.time()
for i in range(1, 100001):
str1 = str1 + test_str
str2 = str2 + test_str
if i % 20000 == 0:
print("time(sec) => ".format(time.time() - start))
start = time.time()
Constant processing time
time(sec) => 0.013324975967407227
time(sec) => 0.020363807678222656
time(sec) => 0.009979963302612305
time(sec) => 0.01744699478149414
time(sec) => 0.0227658748626709
Inexplicably, assigning a concatenated string to another variable makes the process slower and slower.
test_str = "abcdefghijklmn123456789"
str1 = ""
str2 = ""
start = time.time()
for i in range(1, 100001):
str1 = str1 + test_str
# str2 = str2 + test_str
# ↓
str2 = str1
if i % 20000 == 0:
print("time(sec) => ".format(time.time() - start))
start = time.time()
Processing time will be delayed.
time(sec) => 0.36466407775878906
time(sec) => 1.105351209640503
time(sec) => 2.6467738151550293
time(sec) => 5.891657829284668
time(sec) => 9.266698360443115
Both python2 and python3 give the same result.
why?????
python string-concatenation
python string-concatenation
New contributor
New contributor
New contributor
asked 2 hours ago
uma66uma66
362
362
New contributor
New contributor
3
Because you are using a quadratic time algorithm. The interpreter is able to optimize this, but only in some cases. You should not rely on that, and instead, use a linear time algorithm (generally, append to a list then''.join
)
– juanpa.arrivillaga
2 hours ago
As a result of verifying it according to @shadowranger answer, The cause was found. If only string concatenation is used, the ids of str1 and str2 remain the same, but if str1 is assigned to str2, the id of str1 changes each time. In other words, it is slowed by str1 being allocated every time.
– uma66
1 hour ago
add a comment |
3
Because you are using a quadratic time algorithm. The interpreter is able to optimize this, but only in some cases. You should not rely on that, and instead, use a linear time algorithm (generally, append to a list then''.join
)
– juanpa.arrivillaga
2 hours ago
As a result of verifying it according to @shadowranger answer, The cause was found. If only string concatenation is used, the ids of str1 and str2 remain the same, but if str1 is assigned to str2, the id of str1 changes each time. In other words, it is slowed by str1 being allocated every time.
– uma66
1 hour ago
3
3
Because you are using a quadratic time algorithm. The interpreter is able to optimize this, but only in some cases. You should not rely on that, and instead, use a linear time algorithm (generally, append to a list then
''.join
)– juanpa.arrivillaga
2 hours ago
Because you are using a quadratic time algorithm. The interpreter is able to optimize this, but only in some cases. You should not rely on that, and instead, use a linear time algorithm (generally, append to a list then
''.join
)– juanpa.arrivillaga
2 hours ago
As a result of verifying it according to @shadowranger answer, The cause was found. If only string concatenation is used, the ids of str1 and str2 remain the same, but if str1 is assigned to str2, the id of str1 changes each time. In other words, it is slowed by str1 being allocated every time.
– uma66
1 hour ago
As a result of verifying it according to @shadowranger answer, The cause was found. If only string concatenation is used, the ids of str1 and str2 remain the same, but if str1 is assigned to str2, the id of str1 changes each time. In other words, it is slowed by str1 being allocated every time.
– uma66
1 hour ago
add a comment |
1 Answer
1
active
oldest
votes
In general, the Python language standard makes no guarantees here; in fact, as defined, strings are immutable and what you're doing should bite you either way, as you've written a form of Schlemiel the Painter's algorithm.
But in the first case, as an implementation detail, CPython (the reference interpreter) will help you out, and concatenate a string in place (technically violating the immutability guarantee) under some fairly specific conditions that allow it to adhere to the spirit of the immutability rules. The most important condition is that the string being concatenated must be referenced in only one place (if it wasn't, the other reference would change in place, violating the appearance of str
being immutable). By assigning str2 = str1
after each concatenation, you guarantee there are two references when you concatenate, so a new str
must be made by every concatenation to preserve the apparent immutability of strings. That means more memory allocation and deallocation, more (and progressively increasing) memory copies, etc.
Note that relying on this optimization is explicitly discouraged in PEP 8, the Python style guide:
Code should be written in a way that does not disadvantage other implementations of Python (PyPy, Jython, IronPython, Cython, Psyco, and such).
For example, do not rely on CPython's efficient implementation of in-place string concatenation for statements in the form
a += b
ora = a + b
. This optimization is fragile even in CPython (it only works for some types) and isn't present at all in implementations that don't use refcounting. In performance sensitive parts of the library, the''.join()
form should be used instead. This will ensure that concatenation occurs in linear time across various implementations.
The note about "only works for some types" is important. This optimization only applies to str
; in Python 2 it doesn't work on unicode
(even though Python 3's str
is based on the implementation of Python 2's unicode
), and in Python 3 it doesn't work on bytes
(which are similar to Python 2's str
under the hood).
This was exactly what I suspected.
– Tom Karzes
2 hours ago
"so a new str must be made by every concatenation to preserve the apparent immutability of strings." => This was exactly the cause. Thank you!!
– uma66
1 hour ago
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
uma66 is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f56086479%2fwhy-is-it-so-slow-when-assigning-a-concatenated-string-to-a-variable-in-python%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
In general, the Python language standard makes no guarantees here; in fact, as defined, strings are immutable and what you're doing should bite you either way, as you've written a form of Schlemiel the Painter's algorithm.
But in the first case, as an implementation detail, CPython (the reference interpreter) will help you out, and concatenate a string in place (technically violating the immutability guarantee) under some fairly specific conditions that allow it to adhere to the spirit of the immutability rules. The most important condition is that the string being concatenated must be referenced in only one place (if it wasn't, the other reference would change in place, violating the appearance of str
being immutable). By assigning str2 = str1
after each concatenation, you guarantee there are two references when you concatenate, so a new str
must be made by every concatenation to preserve the apparent immutability of strings. That means more memory allocation and deallocation, more (and progressively increasing) memory copies, etc.
Note that relying on this optimization is explicitly discouraged in PEP 8, the Python style guide:
Code should be written in a way that does not disadvantage other implementations of Python (PyPy, Jython, IronPython, Cython, Psyco, and such).
For example, do not rely on CPython's efficient implementation of in-place string concatenation for statements in the form
a += b
ora = a + b
. This optimization is fragile even in CPython (it only works for some types) and isn't present at all in implementations that don't use refcounting. In performance sensitive parts of the library, the''.join()
form should be used instead. This will ensure that concatenation occurs in linear time across various implementations.
The note about "only works for some types" is important. This optimization only applies to str
; in Python 2 it doesn't work on unicode
(even though Python 3's str
is based on the implementation of Python 2's unicode
), and in Python 3 it doesn't work on bytes
(which are similar to Python 2's str
under the hood).
This was exactly what I suspected.
– Tom Karzes
2 hours ago
"so a new str must be made by every concatenation to preserve the apparent immutability of strings." => This was exactly the cause. Thank you!!
– uma66
1 hour ago
add a comment |
In general, the Python language standard makes no guarantees here; in fact, as defined, strings are immutable and what you're doing should bite you either way, as you've written a form of Schlemiel the Painter's algorithm.
But in the first case, as an implementation detail, CPython (the reference interpreter) will help you out, and concatenate a string in place (technically violating the immutability guarantee) under some fairly specific conditions that allow it to adhere to the spirit of the immutability rules. The most important condition is that the string being concatenated must be referenced in only one place (if it wasn't, the other reference would change in place, violating the appearance of str
being immutable). By assigning str2 = str1
after each concatenation, you guarantee there are two references when you concatenate, so a new str
must be made by every concatenation to preserve the apparent immutability of strings. That means more memory allocation and deallocation, more (and progressively increasing) memory copies, etc.
Note that relying on this optimization is explicitly discouraged in PEP 8, the Python style guide:
Code should be written in a way that does not disadvantage other implementations of Python (PyPy, Jython, IronPython, Cython, Psyco, and such).
For example, do not rely on CPython's efficient implementation of in-place string concatenation for statements in the form
a += b
ora = a + b
. This optimization is fragile even in CPython (it only works for some types) and isn't present at all in implementations that don't use refcounting. In performance sensitive parts of the library, the''.join()
form should be used instead. This will ensure that concatenation occurs in linear time across various implementations.
The note about "only works for some types" is important. This optimization only applies to str
; in Python 2 it doesn't work on unicode
(even though Python 3's str
is based on the implementation of Python 2's unicode
), and in Python 3 it doesn't work on bytes
(which are similar to Python 2's str
under the hood).
This was exactly what I suspected.
– Tom Karzes
2 hours ago
"so a new str must be made by every concatenation to preserve the apparent immutability of strings." => This was exactly the cause. Thank you!!
– uma66
1 hour ago
add a comment |
In general, the Python language standard makes no guarantees here; in fact, as defined, strings are immutable and what you're doing should bite you either way, as you've written a form of Schlemiel the Painter's algorithm.
But in the first case, as an implementation detail, CPython (the reference interpreter) will help you out, and concatenate a string in place (technically violating the immutability guarantee) under some fairly specific conditions that allow it to adhere to the spirit of the immutability rules. The most important condition is that the string being concatenated must be referenced in only one place (if it wasn't, the other reference would change in place, violating the appearance of str
being immutable). By assigning str2 = str1
after each concatenation, you guarantee there are two references when you concatenate, so a new str
must be made by every concatenation to preserve the apparent immutability of strings. That means more memory allocation and deallocation, more (and progressively increasing) memory copies, etc.
Note that relying on this optimization is explicitly discouraged in PEP 8, the Python style guide:
Code should be written in a way that does not disadvantage other implementations of Python (PyPy, Jython, IronPython, Cython, Psyco, and such).
For example, do not rely on CPython's efficient implementation of in-place string concatenation for statements in the form
a += b
ora = a + b
. This optimization is fragile even in CPython (it only works for some types) and isn't present at all in implementations that don't use refcounting. In performance sensitive parts of the library, the''.join()
form should be used instead. This will ensure that concatenation occurs in linear time across various implementations.
The note about "only works for some types" is important. This optimization only applies to str
; in Python 2 it doesn't work on unicode
(even though Python 3's str
is based on the implementation of Python 2's unicode
), and in Python 3 it doesn't work on bytes
(which are similar to Python 2's str
under the hood).
In general, the Python language standard makes no guarantees here; in fact, as defined, strings are immutable and what you're doing should bite you either way, as you've written a form of Schlemiel the Painter's algorithm.
But in the first case, as an implementation detail, CPython (the reference interpreter) will help you out, and concatenate a string in place (technically violating the immutability guarantee) under some fairly specific conditions that allow it to adhere to the spirit of the immutability rules. The most important condition is that the string being concatenated must be referenced in only one place (if it wasn't, the other reference would change in place, violating the appearance of str
being immutable). By assigning str2 = str1
after each concatenation, you guarantee there are two references when you concatenate, so a new str
must be made by every concatenation to preserve the apparent immutability of strings. That means more memory allocation and deallocation, more (and progressively increasing) memory copies, etc.
Note that relying on this optimization is explicitly discouraged in PEP 8, the Python style guide:
Code should be written in a way that does not disadvantage other implementations of Python (PyPy, Jython, IronPython, Cython, Psyco, and such).
For example, do not rely on CPython's efficient implementation of in-place string concatenation for statements in the form
a += b
ora = a + b
. This optimization is fragile even in CPython (it only works for some types) and isn't present at all in implementations that don't use refcounting. In performance sensitive parts of the library, the''.join()
form should be used instead. This will ensure that concatenation occurs in linear time across various implementations.
The note about "only works for some types" is important. This optimization only applies to str
; in Python 2 it doesn't work on unicode
(even though Python 3's str
is based on the implementation of Python 2's unicode
), and in Python 3 it doesn't work on bytes
(which are similar to Python 2's str
under the hood).
edited 2 hours ago
answered 2 hours ago
ShadowRangerShadowRanger
65.4k664103
65.4k664103
This was exactly what I suspected.
– Tom Karzes
2 hours ago
"so a new str must be made by every concatenation to preserve the apparent immutability of strings." => This was exactly the cause. Thank you!!
– uma66
1 hour ago
add a comment |
This was exactly what I suspected.
– Tom Karzes
2 hours ago
"so a new str must be made by every concatenation to preserve the apparent immutability of strings." => This was exactly the cause. Thank you!!
– uma66
1 hour ago
This was exactly what I suspected.
– Tom Karzes
2 hours ago
This was exactly what I suspected.
– Tom Karzes
2 hours ago
"so a new str must be made by every concatenation to preserve the apparent immutability of strings." => This was exactly the cause. Thank you!!
– uma66
1 hour ago
"so a new str must be made by every concatenation to preserve the apparent immutability of strings." => This was exactly the cause. Thank you!!
– uma66
1 hour ago
add a comment |
uma66 is a new contributor. Be nice, and check out our Code of Conduct.
uma66 is a new contributor. Be nice, and check out our Code of Conduct.
uma66 is a new contributor. Be nice, and check out our Code of Conduct.
uma66 is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f56086479%2fwhy-is-it-so-slow-when-assigning-a-concatenated-string-to-a-variable-in-python%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
3
Because you are using a quadratic time algorithm. The interpreter is able to optimize this, but only in some cases. You should not rely on that, and instead, use a linear time algorithm (generally, append to a list then
''.join
)– juanpa.arrivillaga
2 hours ago
As a result of verifying it according to @shadowranger answer, The cause was found. If only string concatenation is used, the ids of str1 and str2 remain the same, but if str1 is assigned to str2, the id of str1 changes each time. In other words, it is slowed by str1 being allocated every time.
– uma66
1 hour ago