How to Join Member's Area Private Library Search Today's Topics p Login
Main Forums Discussion Tech Talk Mature Content Archives
   Nav Win
 Discussion
 Announcements & Links
 The Forums are back
 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Follow us on Facebook

 Moderated by: Elizabeth   (Admins )

 
User Options
Format for Better Printing EMail to a Friend Not Available
Admin Print Send ECard
Passions in Poetry

The Forums are back

 Post A Reply Post New Topic   Go to the Next Oldest/Previous Topic Return to Topic Page Go to the Next Newest Topic 
Ron
Administrator
Member Rara Avis
since 05-19-99
Posts 9708
Michigan, US


0 posted 11-11-2005 05:15 AM       View Profile for Ron   Email Ron   Edit/Delete Message      Find Poems   Click to visit Ron's Home Page   View IP for Ron

'Twas an interesting evening.

The forums went down Thursday night, about 10:38 EST as best I can tell, and remained down until the wee hours of the morning (see time on this post). We can all thank Nan for letting me know there was a problem (though she blamed the problem on me "doing something"), because we would have lost a LOT of data if I hadn't shut the forums down immediately.

As it is, I think we lost exactly one thread, from Open #36. It was number 003109, but I don't know the title or the author. We owe someone an apology, but I don't know who?

The short story is that we ran out of disk space on the server.

That has the potential to do a lot of damage, because the server keeps right on trucking, trying to write data to a disk that won't accept it. Every time it tries to update an existing data file, like thread 003109, it instead mangles it horribly. That's why shutting down the forum right away was so incredibly important. We were darn lucky Nan emailed me, and darn lucky I was still at the keyboard to get her message.

The long story is a bit more technical.

Every time someone responds to an existing thread, the server saves the post to a database file and then updates the HTML page with the same post. Essentially, everything is saved twice, once in a database and once as a web page. The big exception to that rule is our Archives section. Because those typically get a lot less traffic, the system only builds an HTML page when someone actually clicks on a link. Most of the data is thus only stored once, instead of twice, and that savings equates to massive gigabytes of disk space.

Would you believe someone went through the forums this week and clicked on every single thread in every single archive? That's nearly 200,000 clicks, each one of which created a new HTML page. Ergo, we very quickly ran out of disk space.

Ironically, when a Linux box is out of disk, it's difficult to do much of anything. Even deleting a file requires a few bytes of swap space, and the bigger the file you want to delete, the more swap space is needed. It took me almost two hours of zapping little files to get the disk from 100% used to 99% used, giving me enough free disk to actually be able to do something useful. Getting rid of all those extra HTML files in the Archives took another hour, but quickly took the disk down to only 52% used (which is about where we were last week before our mysterious culprit went click-crazy in the Archives).

Once I found some elbow room, I had to rebuild virtually every single file in the forums. I'm about four hours into the process as I write this, with probably another two hours to go. Yea, it's been a real interesting night.

So, who went crazy in the Archives and created almost 200K files in the process?

Blame it on Google.

p.s. Please, everyone, keep your eyes open for potentially corrupt data files or web pages. I *think* everything is fixed, but a few extra sets of eyes sure won't hurt.
Kit McCallum
Administrator
Member Laureate
since 04-30-2000
Posts 16920
Ontario, Canada


1 posted 11-11-2005 05:40 AM       View Profile for Kit McCallum   Email Kit McCallum   Edit/Delete Message      Find Poems  View IP for Kit McCallum

Thank you Nan!  And thanks for working through it Ron.

Wow, that's one busy reader, lol. Maybe check for the person with a sore wrist and click-finger ... that's a LOT of clicking, lol.
Poet deVine
Administrator
Member Empyrean
since 05-26-99
Posts 25869
Hurricane Alley


2 posted 11-11-2005 05:56 AM       View Profile for Poet deVine   Email Poet deVine   Edit/Delete Message      Find Poems  View IP for Poet deVine

Yikes! Sounds like someone was busy..or created a program to open files in the archives?

Thank you Nan for being 'awake'!! Whew!
Sunshine
Administrator
Member Caelestus
since 06-25-99
Posts 67715
Listening to every heart


3 posted 11-11-2005 06:10 AM       View Profile for Sunshine   Email Sunshine   Edit/Delete Message      Find Poems   Click to visit Sunshine's Home Page   View IP for Sunshine

Thank you, both!

Sunshine
Administrator
Member Caelestus
since 06-25-99
Posts 67715
Listening to every heart


4 posted 11-11-2005 06:49 AM       View Profile for Sunshine   Email Sunshine   Edit/Delete Message      Find Poems   Click to visit Sunshine's Home Page   View IP for Sunshine

Ron, I think I see where we lost a few pages.

I went to Prose, clicked on Copperbell's post, clicked on Larry's "Taco Bell" link, and poof - no page to be found.

Went into Archives #3 [I think], WhtDove's post of 10/29/05 was at the top, went to click in to page 2 to see who had responded...Poof - no page to be found.

There may still be a few bugs?
nakdthoughts
Member Laureate
since 10-29-2000
Posts 19275
Between the Lines


5 posted 11-11-2005 06:58 AM       View Profile for nakdthoughts   Email nakdthoughts   Edit/Delete Message      Find Poems  View IP for nakdthoughts

wow..I can't even imagine who would have the time to read all those in the archives. I have enough trouble finding the time to read what is recent.

Maybe it was someone new who wanted to "catch up"
Thank you Nan for paying such good attention and thank you Ron for this site and ALL you do.

M
Nan
Administrator
Member Seraphic
since 05-20-99
Posts 24426
Cape Cod Massachusetts USA


6 posted 11-11-2005 07:58 AM       View Profile for Nan   Email Nan   Edit/Delete Message      Find Poems   Click to visit Nan's Home Page   View IP for Nan

I can't take all the credit... I actually was getting into the threads with no difficulty.  BUT - It was a rare moment where I happened to be on both MSN and Yahoo, as I was building webpages and needed to be able to message.

If it weren't for Mysteria, Sunshine, Vicky, and Martie all reporting they were unable to access PiP all at the same time - coupled with my inability to FTP into the site - I wouldn't have known either.

Teamwork - Yeah!
Larry C
Deputy Moderator 1 Tour
Member Patricius
since 09-10-2001
Posts 10765
United States


7 posted 11-11-2005 09:31 AM       View Profile for Larry C   Email Larry C   Edit/Delete Message      Find Poems   Click to visit Larry C's Home Page   View IP for Larry C

So THAT'S what you do with your spare time! Wow! You are the most dedicated man I know. Thanks Ron. Click crazy for sure. Glad it wasn't me.
Nightshade
Deputy Moderator 5 ToursDeputy Moderator 1 TourDeputy Moderator 1 TourDeputy Moderator 1 Tour
Member Laureate
since 08-31-2001
Posts 14673
just out of reach


8 posted 11-11-2005 09:34 AM       View Profile for Nightshade   Email Nightshade   Edit/Delete Message      Find Poems   Click to visit Nightshade's Home Page   View IP for Nightshade

I don't quite understand all of the technical stuff .... but a big, big thanks and lots of hugs to all of those who helped get our home up and running again !!
Ron
Administrator
Member Rara Avis
since 05-19-99
Posts 9708
Michigan, US


9 posted 11-11-2005 12:28 PM       View Profile for Ron   Email Ron   Edit/Delete Message      Find Poems   Click to visit Ron's Home Page   View IP for Ron

Karilea, I'll check out those threads (after some coffee), but I'm not sure there's much I can do to fix them.

As for our click-crazy culprit, I guess I was too tired last night to be entirely clear?

The search engines all send out automated programs, called spiders or robots, to "crawl the web" and find pages that will be added to the search database. That's how all our stuff gets into a search engine in the first place. In the past few days, Google's spider (called Googlebot) went through pretty much the entire forum structure, including those critical Archive threads.

This is the first time we've attracted quite so much attention from the spiders, but if my experience on the main site is any measure, it won't be the last. Now that Googlebot knows we have so much spider food hidden in the back rooms, he'll likely become a very regular visitor. I need to slip back into programming mode for a few days and redesign our Archive system so it doesn't create those extra pages and fill up the disk. There are several possible directions I still need to investigate, but it obviously just became a priority. I'm getting much too old for these all-nighters.  

Incidentally, a few years ago I had a little fun with everyone as we approached our one million post milestone. Last night, as I was trudging through the database, I realized we passed our two million milestone some time in the past week or two, with nary a balloon or festive whoot in sight.

Whoot, whoot?  


serenity blaze
Member Empyrean
since 02-02-2000
Posts 28839


10 posted 11-11-2005 12:32 PM       View Profile for serenity blaze   Email serenity blaze   Edit/Delete Message      Find Poems  View IP for serenity blaze

You are always very clear.



Doesn't mean I know what the heck you are talking about though.

(But I'll bet I could still beat you at game of "Break Out.")



and yep, whoot-whoot!
Mysteria
Deputy Moderator 10 ToursDeputy Moderator 10 ToursDeputy Moderator 10 ToursDeputy Moderator 10 ToursDeputy Moderator 1 TourDeputy Moderator 1 TourDeputy Moderator 1 Tour
Member Laureate
since 03-07-2001
Posts 19652
British Columbia, Canada


11 posted 11-11-2005 01:42 PM       View Profile for Mysteria   Email Mysteria   Edit/Delete Message      Find Poems  View IP for Mysteria

Never admit you are too old for an all-nighters, never!  Now about that Whoot-whoot?  Must be another American term?

Is that like our Canadian whoo-hooo?

Poor you!  I mean that sincerely. I was talking to Nan, and poof the lights went out in Passions just like that!  Sure glad you have a supply of fuses there.
sandgrain
Member Elite
since 09-21-1999
Posts 3657
Sycamore, IL, USA


12 posted 11-11-2005 04:05 PM       View Profile for sandgrain   Email sandgrain   Edit/Delete Message      Find Poems  View IP for sandgrain

It's really hard to remember life without PIP once it becomes a daily watering hole.  Many thanks, Ron, to you and all the others who keep PIP family blue pages available and so organized for all of us.

Do you suppose someone actually read all those or just fell asleep clicking away?

  Rae
Christopher
Moderator
Member Rara Avis
since 08-02-99
Posts 9130
Purgatorial Incarceration


13 posted 11-11-2005 05:05 PM       View Profile for Christopher   Email Christopher   Edit/Delete Message      Find Poems  View IP for Christopher

definitely wasn't me... i don't have DSL installed at our new house yet. so no pointing fingers!
iliana
Member Patricius
since 12-05-2003
Posts 13488
USA


14 posted 11-11-2005 05:31 PM       View Profile for iliana   Email iliana   Edit/Delete Message      Find Poems  View IP for iliana

Thanks, Ron.  And, thanks to Nan, too.  Wow, Ron, I cannot believe the time you devote to this place.  We are all so grateful.   ....jo
Ron
Administrator
Member Rara Avis
since 05-19-99
Posts 9708
Michigan, US


15 posted 11-11-2005 06:58 PM       View Profile for Ron   Email Ron   Edit/Delete Message      Find Poems   Click to visit Ron's Home Page   View IP for Ron

quote:
Is that like our Canadian whoo-hooo?

http://www.woot.com/

It doesn't quite rhyme with eBay, Sharon, but it's close.

They sell one and only one thing a day, usually at a good price. I bought a wireless computer speaker just the other day, so I can take my music out into the yard next Spring. It's the kind of place I suspect you would love.

Unfortunately, I'm probably teasing you, 'cause I don't think they'll ship to Vancouver.


Mysteria
Deputy Moderator 10 ToursDeputy Moderator 10 ToursDeputy Moderator 10 ToursDeputy Moderator 10 ToursDeputy Moderator 1 TourDeputy Moderator 1 TourDeputy Moderator 1 Tour
Member Laureate
since 03-07-2001
Posts 19652
British Columbia, Canada


16 posted 11-12-2005 01:25 AM       View Profile for Mysteria   Email Mysteria   Edit/Delete Message      Find Poems  View IP for Mysteria

I have literally died and gone to heaven!  Been in that Woot for hours LOL, but then you knew that I would be right?  If they don't ship here, I can send it to the girls - there is always a way Ron, always a way     Thanks for the link, I love it!  You find the neatest places and things dontca?  Still laughing at November 4th item for grabs - gotta get me a bag of that for sure!

Oh, and I never heard of a woot before or a whoot either LOL
Ron
Administrator
Member Rara Avis
since 05-19-99
Posts 9708
Michigan, US


17 posted 11-14-2005 06:15 AM       View Profile for Ron   Email Ron   Edit/Delete Message      Find Poems   Click to visit Ron's Home Page   View IP for Ron

I've changed the way our software handles Archive files, a task which turned out to be a lot easier than I anticipated. Henceforth, all the Archive threads will by dynamic cgi pages instead of static HTML ones. Unfortunately, this will break the few old links to those static pages, but is a necessary cost to protecting us from the search engine spiders.


Kit McCallum
Administrator
Member Laureate
since 04-30-2000
Posts 16920
Ontario, Canada


18 posted 11-14-2005 06:40 AM       View Profile for Kit McCallum   Email Kit McCallum   Edit/Delete Message      Find Poems  View IP for Kit McCallum

I'm glad you found a way to handle the archives without taking on a few more all-nighters Ron. Let the spiders and happy clickers have at it now, lol!

But ... we missed the countdown???



TWO million posts???



Woot!  Whoot!     Whoo Hoo!!!

and a big wide-eyed WOW!!! Unbelievable numbers!!!

Marge Tindal
Deputy Moderator 5 ToursDeputy Moderator 1 TourDeputy Moderator 1 TourDeputy Moderator 1 Tour
Member Empyrean
since 11-06-1999
Posts 43042
Florida's Foreverly Shores


19 posted 11-14-2005 12:00 PM       View Profile for Marge Tindal   Email Marge Tindal   Edit/Delete Message      Find Poems   Click to visit Marge Tindal's Home Page   View IP for Marge Tindal


quote:
Unfortunately, this will break the few old links to those static pages, but is a necessary cost to protecting us from the search engine spiders.

HUH ?
Christioned
Member
since 01-15-2004
Posts 93
Huddling in the dark


20 posted 11-14-2005 08:18 PM       View Profile for Christioned   Email Christioned   Edit/Delete Message      Find Poems  View IP for Christioned

Thank you so much Ron for keeping our happy little sanctuary up and running. Thank you also to Nan and everyone else who found out that there was a problem. Don't know what I'd do if Pip wasn't here to come back to.
Also ya hoo on 2 millionth post. Hope that we will make it 3 millionth and many more.  

"See my wrists;
I feel your pain."-Creed

icebox
Member Elite
since 05-03-2003
Posts 4246
in the shadows


21 posted 11-15-2005 02:10 PM       View Profile for icebox   Email icebox   Edit/Delete Message      Find Poems  View IP for icebox

Too many clicks!?!?!!!?

...uh...so you mean I should stop writing on the Men's Room walls: "For a good time click http://www.piptalk.com/ " ?????

(hey?  thanks once again for the almost infinite labor of maitaining this place!)
serenity blaze
Member Empyrean
since 02-02-2000
Posts 28839


22 posted 08-28-2007 07:21 PM       View Profile for serenity blaze   Email serenity blaze   Edit/Delete Message      Find Poems  View IP for serenity blaze

Hey Ron! :waving:

I believe my memory really is serving me better these days...



I just thought I'd bump this because I found it so very interesting in light of recent events.

I'll still write the letter folks.
Susan Caldwell
Member Rara Avis
since 12-27-2002
Posts 8464
Florida


23 posted 08-28-2007 08:21 PM       View Profile for Susan Caldwell   Edit/Delete Message      Find Poems  View IP for Susan Caldwell

Wow, that is interesting Karen..

"too bad ignorance isn't painful"
~Unknown~

 
 Post A Reply Post New Topic   Go to the Next Oldest/Previous Topic Return to Topic Page Go to the Next Newest Topic 
All times are ET (US) Top
  User Options
>> Discussion >> Announcements & Links >> The Forums are back Format for Better Printing EMail to a Friend Not Available
Print Send ECard

 

pipTalk Home Page | Main Poetry Forums

How to Join | Member's Area / Help | Private Library | Search | Contact Us | Today's Topics | Login
Discussion | Tech Talk | Archives | Sanctuary



© Passions in Poetry and netpoets.com 1998-2013
All Poetry and Prose is copyrighted by the individual authors