'open file' Thread RSS Feed

open file

Mon, 09 Feb 2004 00:39:16 -0700

I'm trying to open a file but when the server runs the following it dies at the following line. It doesn't print out any error, just stops all togeather. Is file size an issue, its a 5Mb file.

open(FILE,"$basepath/reminders_location.txt") || print "Unable to open file: $!<br>\n";

Thanks,
David

Re: open file

Mon, 09 Feb 2004 04:58:59 -0700

: I'm trying to open a file but when the server runs the following it
: dies at the following line. It doesn't print out any error, just
: stops all togeather. Is file size an issue, its a 5Mb file.
:

open(FILE,"$basepath/reminders_location.txt") || print "Unable to open file: $!\n";

:
Have you tried running the script from the shell? Remember that errors go to STDERR, not STDOUT, and STDERR is not directed to CGI output. I find it very, very unlikely that file size comes into it, but file permissions on the other had are possibly the issue - the user the script runs has will have the permissions to access the file, right? Also try giving an access method:-

open(FILE,"< $basepath/reminders_location.txt") || print "Unable to open file: $!\n";

To read, for example.

Jonathan

###
for(74,117,115,116){$::a.=chr};(($_.='qwertyui')&&
(tr/yuiqwert/her anot/))for($::b);for($::c){$_.=$^X;
/(p.{2}l)/;$_=$1}$::b=~/(..)$/;print("$::a$::b $::c hack$1.");

Re: open file

Mon, 09 Feb 2004 07:35:17 -0700

: : I'm trying to open a file but when the server runs the following it
: : dies at the following line. It doesn't print out any error, just
: : stops all togeather. Is file size an issue, its a 5Mb file.
: :

open(FILE,"$basepath/reminders_location.txt") || print "Unable to open file: $!\n";

: :
: Have you tried running the script from the shell? Remember that errors go to STDERR, not STDOUT, and STDERR is not directed to CGI output. I find it very, very unlikely that file size comes into it, but file permissions on the other had are possibly the issue - the user the script runs has will have the permissions to access the file, right? Also try giving an access method:-
:
: open(FILE,"< $basepath/reminders_location.txt") || print "Unable to open file: $!\n";
:
: To read, for example.
:
: Jonathan
:
: ###
: for(74,117,115,116){$::a.=chr};(($_.='qwertyui')&&
: (tr/yuiqwert/her anot/))for($::b);for($::c){$_.=$^X;
: /(p.{2}l)/;$_=$1}$::b=~/(..)$/;print("$::a$::b $::c hack$1.");
:
:

I dropped the file down to 3500 lines of text from 9576 and it loads fine. Tried it at 4000 and still wasn't loading. Its a 1.7 mb file where the original is 5.3 mb. And its another one of those cross server problems where it is working fine on one server but not the other heh. Gotta love it!

David

Re: open file

Mon, 09 Feb 2004 09:44:59 -0700

: I dropped the file down to 3500 lines of text from 9576 and it loads fine. Tried it at 4000 and still wasn't loading. Its a 1.7 mb file where the original is 5.3 mb. And its another one of those cross server problems where it is working fine on one server but not the other heh. Gotta love it!
:
Strange, very strange. Open should in theory just be obtaining a file handle - are you sure it's not actually failing at a later stage (e.g. when you attempt to read the entire file into an array)? If you're sure that the open line is the problem then you can always try using the sysopen function instead. That clears out the way any kinky stuff Perl is doing that causes the problem (I'm not familiar with the P5 I/O layer, so I'm not quite sure what it would do though...).

Jonathan

###
for(74,117,115,116){$::a.=chr};(($_.='qwertyui')&&
(tr/yuiqwert/her anot/))for($::b);for($::c){$_.=$^X;
/(p.{2}l)/;$_=$1}$::b=~/(..)$/;print("$::a$::b $::c hack$1.");

Re: open file

Mon, 09 Feb 2004 10:06:31 -0700

This message was edited by davidrtg at 2004-2-9 10:20:9

: : I dropped the file down to 3500 lines of text from 9576 and it loads fine. Tried it at 4000 and still wasn't loading. Its a 1.7 mb file where the original is 5.3 mb. And its another one of those cross server problems where it is working fine on one server but not the other heh. Gotta love it!
: :
: Strange, very strange. Open should in theory just be obtaining a file handle - are you sure it's not actually failing at a later stage (e.g. when you attempt to read the entire file into an array)? If you're sure that the open line is the problem then you can always try using the sysopen function instead. That clears out the way any kinky stuff Perl is doing that causes the problem (I'm not familiar with the P5 I/O layer, so I'm not quite sure what it would do though...).
:
: Jonathan
:
: ###
: for(74,117,115,116){$::a.=chr};(($_.='qwertyui')&&
: (tr/yuiqwert/her anot/))for($::b);for($::c){$_.=$^X;
: /(p.{2}l)/;$_=$1}$::b=~/(..)$/;print("$::a$::b $::c hack$1.");
:
:

Ahhhh.. its not that line, but the line that reads in into the array!

(@location) = <FILE>;

Any ideas why an array would be having problems with that size of information?

EDIT: After some research I found that it could be a system memory thing. I'm going to see if I can figure a way to read each line in <FILE> and work with it that way instead of loading the whole thing into the array.

EDIT: That did it, just did while(<FILE>) and all is well!

Thanks once again Jonathan!
David

Re: open file

Mon, 09 Feb 2004 10:20:54 -0700

: Ahhhh.. its not that line, but the line that reads in into the
: array!

: (@location) = <FILE>;

: Any ideas why an array would be having problems with that size of
: information?
:
Probably because of the quantity of it. If you're loading a file with 10,000 lines that's about 5 MB, it's going to require more than 5 MB of RAM. Why? Perl internals. As a vast simplification, all individual items are stored in SVs (scalar values). An AV (array value) is implemented as a collection of SVs, probably arranged in some kind of tree structure to make array index lookup logarithmic or better (I don't know that, they could have something better in there). Each SV will have some overhead (data about the length of the value, reference count, etc) and the AV will have a little too. Perl is pretty efficient in general, but the runtime just needs that extra data to keep track of memory usage and the like. Still, I would *hope* it'd be able to manage this much even though it's not ideal.

You're basically asking for a lot of memory, and it sounds like on a certain system it's just not willing to let you have it. Possibly the server has one of those CGI resource limiting things set up to stop scripts hogging lots of resources. Possibly there's something else getting in the way.

Is there any reason you can not do it like this:-

while ($line = <FILE>) {
    # Do stuff with $line
}

Or don't put $line and the data will be in $_. That would probably be your best way to do it. This way all the data ain't read in at once, the file is just read one line at a time.

Jonathan

###
for(74,117,115,116){$::a.=chr};(($_.='qwertyui')&&
(tr/yuiqwert/her anot/))for($::b);for($::c){$_.=$^X;
/(p.{2}l)/;$_=$1}$::b=~/(..)$/;print("$::a$::b $::c hack$1.");

Re: open file

Mon, 09 Feb 2004 15:10:02 -0700

: : Ahhhh.. its not that line, but the line that reads in into the
: : array!
:

: (@location) = <FILE>;

: : Any ideas why an array would be having problems with that size of
: : information?
: :
: Probably because of the quantity of it. If you're loading a file with 10,000 lines that's about 5 MB, it's going to require more than 5 MB of RAM. Why? Perl internals. As a vast simplification, all individual items are stored in SVs (scalar values). An AV (array value) is implemented as a collection of SVs, probably arranged in some kind of tree structure to make array index lookup logarithmic or better (I don't know that, they could have something better in there). Each SV will have some overhead (data about the length of the value, reference count, etc) and the AV will have a little too. Perl is pretty efficient in general, but the runtime just needs that extra data to keep track of memory usage and the like. Still, I would *hope* it'd be able to manage this much even though it's not ideal.
:
: You're basically asking for a lot of memory, and it sounds like on a certain system it's just not willing to let you have it. Possibly the server has one of those CGI resource limiting things set up to stop scripts hogging lots of resources. Possibly there's something else getting in the way.
:
: Is there any reason you can not do it like this:-
:
:

while ($line = <FILE>) {
:     # Do stuff with $line
: }

:
: Or don't put $line and the data will be in $_. That would probably be your best way to do it. This way all the data ain't read in at once, the file is just read one line at a time.
:
: Jonathan
:
: ###
: for(74,117,115,116){$::a.=chr};(($_.='qwertyui')&&
: (tr/yuiqwert/her anot/))for($::b);for($::c){$_.=$^X;
: /(p.{2}l)/;$_=$1}$::b=~/(..)$/;print("$::a$::b $::c hack$1.");
:
:

Yah, i'm thinking its that there is a restriction on the ammount of memory I can use on the server.
Got it working with the while(<FILE>)!

Thanks again,
David