When I removed some unused IP's on eth0 my whole network broke, Help wanted!

17 posts / 0 new
Last post
#1 Wed, 11/15/2006 - 19:31
Blueforce

When I removed some unused IP's on eth0 my whole network broke, Help wanted!

Hi everyone,

Looks like I messed up big time! Postfix was sometimes trying to use a IP registred on eth0 that caused delivery to fail due to recipeint couldn't resolve a hostname from that IP(This IP was not connected to any domain yet). Ok, saw some mails from a cutomer in postfix que that couldn't get delivered due to this. An me "the genius" thought that I just could un-activate that specific IP from the active network interfaces.... ticked the box for that interface, and hit "un-activate" button (don't actually remeber the exact button name)... Yepp I really did un-activate it AND the whole network(I think) at the same time, and now I have an unreachable server. Have ANYONE some ideas how I should do to reach the server so I can do a restart, all interfaces ar still in the active at bootup. But I can't reach the server to do this!

Any help is really appreciated.

This is really not good becauce earlier today we hade one of our biggest ISP fiber network breaking down and their redundance didn't kick in, leaving about 30000 websites/server without internet connection for about 4 hours, including our server. And now this... my really f..... up!! :-(

Our customers do NOT like this!

Please feel free to jump in any time with nice solutions!

Regards

Leif

Sun, 06/07/2009 - 07:01
Joe
Joe's picture

Hi Leif,

If none of your interfaces are active, you're not going to be able to connect via network...there's just no way in. I'll assume you've tried to login via all of them?

If you didn't actually delete it from the "Interfaces activated at boot time" section, then a reboot will bring it all back up, and you can log back in. Most hosting providers offer reboot services 24 hours a day. Some even have a web interface for remote reboots that you can activate yourself.

And making Postfix pick up a different IP probably isn't going to solve the issue you're seeing--reverse resolution is probably the source of that trouble, and unless you system has been delegated to provide reverse resolution for your IPs, having your name server know about the IP and it's name(s) won't do any good. Reverse resolution is usually handled by your hosting provider or ISP by default, but you can usually request delegation (or request they update the name to suit your purposes).

--

Check out the forum guidelines!

Sun, 06/07/2009 - 07:01
Blueforce

Hi Joe

I was afraid that this would be yoyr answer, and yes... have tried all I can think of! I actually didn't think the default network would breake when remowing one other IP(interface). The problem with postfix some times picking up another IP than the default I somehow have to resolv. We have access to our ISP's DNS server to for adding, changing DNS settings and entrys.

I'm thinking of installing a box in our office with FC4 and Virtualmin Pro(same license) for cases like this when I'm messing things up like this. I Think I have asked this before if it's ok to do so, the one in our office is only in use and accessible from internet in cases like this. I can easally install it and maybe restore all domains from backupfiles, I have all domains and all wmin config backups.

It's easy for me to change our DNS to point to our private IP instead, and the changes is instant.

Do you think this ok to do, call it our "mirror" server.

Our Server Hall have 24/7 support for those who pay for it, in other words... we are not, and our suport opens in 5 hours so they can manually restart the box.

I'm anyway thinking of doing the mirror thing if that is by license things. I think this is the best way, I could point all domains to our US server but it's running WHM/Cpanel :-( and all customers have to shage their email usernames and so on...

So do you think I should go for it... I can do some usefull work during the night instead of just wating for our Server Hall to open.

Regards,
Leif

Sun, 06/07/2009 - 07:01
Blueforce

Hi Joe

I was afraid that this would be yoyr answer, and yes... have tried all I can think of! I actually didn't think the default network would breake when remowing one other IP(interface). The problem with postfix some times picking up another IP than the default I somehow have to resolv. We have access to our ISP's DNS server to for adding, changing DNS settings and entrys.

I'm thinking of installing a box in our office with FC4 and Virtualmin Pro(same license) for cases like this when I'm messing things up like this. I Think I have asked this before if it's ok to do so, the one in our office is only in use and accessible from internet in cases like this. I can easally install it and maybe restore all domains from backupfiles, I have all domains and all wmin config backups.

It's easy for me to change our DNS to point to our private IP instead, and the changes is instant.

Do you think this ok to do, call it our "mirror" server.

Our Server Hall have 24/7 support for those who pay for it, in other words... we are not, and our suport opens in 5 hours so they can manually restart the box.

I'm anyway thinking of doing the mirror thing if that is by license things. I think this is the best way, I could point all domains to our US server but it's running WHM/Cpanel :-( and all customers have to shage their email usernames and so on...

So do you think I should go for it... I can do some usefull work during the night instead of just wating for our Server Hall to open.

Regards,
Leif

Sun, 06/07/2009 - 07:01
Joe
Joe's picture

Hey Leif,

Yes, a hot spare is definitely allowed by our license terms. We're generous to a fault. ;-)

The only rule for hot spares and devel servers: The hot spare must have identical domains to your primary or no domains that live longer than 30 days. Our license manager doesn't actually enforce this limitation at the moment, but it will in about a week, so might as well pretend like it's already a fact.

On the network issue, I'll take this opportunity to remind you of our conversation about changing IPs and how to be safe about it here:

http://www.virtualmin.com/forums/message-view?message_id=64004

But I won't pick on you too much for not being as careful this time. I've done the same thing a half dozen times...thinking I wasn't disabling the only interface that was actually working, or the parent of all other interfaces. ;-)

--

Check out the forum guidelines!

Sun, 06/07/2009 - 07:01
Blueforce

Hi Joe

I was afraid that this would be yoyr answer, and yes... have tried all I can think of! I actually didn't think the default network would breake when remowing one other IP(interface). The problem with postfix some times picking up another IP than the default I somehow have to resolv. We have access to our ISP's DNS server to for adding, changing DNS settings and entrys.

I'm thinking of installing a box in our office with FC4 and Virtualmin Pro(same license) for cases like this when I'm messing things up like this. I Think I have asked this before if it's ok to do so, the one in our office is only in use and accessible from internet in cases like this. I can easally install it and maybe restore all domains from backupfiles, I have all domains and all wmin config backups.

It's easy for me to change our DNS to point to our private IP instead, and the changes is instant.

Do you think this ok to do, call it our "mirror" server.

Our Server Hall have 24/7 support for those who pay for it, in other words... we are not, and our suport opens in 5 hours so they can manually restart the box.

I'm anyway thinking of doing the mirror thing if that is by license things. I think this is the best way, I could point all domains to our US server but it's running WHM/Cpanel :-( and all customers have to shage their email usernames and so on...

So do you think I should go for it... I can do some usefull work during the night instead of just wating for our Server Hall to open.

Regards,
Leif

Sun, 06/07/2009 - 07:01
Joe
Joe's picture

Hey Leif,

Yes, a hot spare is definitely allowed by our license terms. We're generous to a fault. ;-)

The only rule for hot spares and devel servers: The hot spare must have identical domains to your primary or no domains that live longer than 30 days. Our license manager doesn't actually enforce this limitation at the moment, but it will in about a week, so might as well pretend like it's already a fact.

On the network issue, I'll take this opportunity to remind you of our conversation about changing IPs and how to be safe about it here:

http://www.virtualmin.com/forums/message-view?message_id=64004

But I won't pick on you too much for not being as careful this time. I've done the same thing a half dozen times...thinking I wasn't disabling the only interface that was actually working, or the parent of all other interfaces. ;-)

--

Check out the forum guidelines!

Sun, 06/07/2009 - 07:01
Blueforce

Ok,

And the domains will be exactly the same, and hopfully I don't breake things on our server for as long as 30 days :-)

And I actually did look at our earlier conversation regarding this... I thought eth0 would still run after inactivating eth0:5 from active list.

Will there be some network issues when restoring domains from the backup to our local server, our local sever will get another public IP than the "real" server.

Leif

Sun, 06/07/2009 - 07:01
Blueforce

Hi Joe

I was afraid that this would be yoyr answer, and yes... have tried all I can think of! I actually didn't think the default network would breake when remowing one other IP(interface). The problem with postfix some times picking up another IP than the default I somehow have to resolv. We have access to our ISP's DNS server to for adding, changing DNS settings and entrys.

I'm thinking of installing a box in our office with FC4 and Virtualmin Pro(same license) for cases like this when I'm messing things up like this. I Think I have asked this before if it's ok to do so, the one in our office is only in use and accessible from internet in cases like this. I can easally install it and maybe restore all domains from backupfiles, I have all domains and all wmin config backups.

It's easy for me to change our DNS to point to our private IP instead, and the changes is instant.

Do you think this ok to do, call it our "mirror" server.

Our Server Hall have 24/7 support for those who pay for it, in other words... we are not, and our suport opens in 5 hours so they can manually restart the box.

I'm anyway thinking of doing the mirror thing if that is by license things. I think this is the best way, I could point all domains to our US server but it's running WHM/Cpanel :-( and all customers have to shage their email usernames and so on...

So do you think I should go for it... I can do some usefull work during the night instead of just wating for our Server Hall to open.

Regards,
Leif

Sun, 06/07/2009 - 07:01
Joe
Joe's picture

Hey Leif,

Yes, a hot spare is definitely allowed by our license terms. We're generous to a fault. ;-)

The only rule for hot spares and devel servers: The hot spare must have identical domains to your primary or no domains that live longer than 30 days. Our license manager doesn't actually enforce this limitation at the moment, but it will in about a week, so might as well pretend like it's already a fact.

On the network issue, I'll take this opportunity to remind you of our conversation about changing IPs and how to be safe about it here:

http://www.virtualmin.com/forums/message-view?message_id=64004

But I won't pick on you too much for not being as careful this time. I've done the same thing a half dozen times...thinking I wasn't disabling the only interface that was actually working, or the parent of all other interfaces. ;-)

--

Check out the forum guidelines!

Sun, 06/07/2009 - 07:01
Blueforce

Ok,

And the domains will be exactly the same, and hopfully I don't breake things on our server for as long as 30 days :-)

And I actually did look at our earlier conversation regarding this... I thought eth0 would still run after inactivating eth0:5 from active list.

Will there be some network issues when restoring domains from the backup to our local server, our local sever will get another public IP than the "real" server.

Leif

Sun, 06/07/2009 - 07:01
Joe
Joe's picture

Hey Leif,

<i>And the domains will be exactly the same, and hopfully I don't breake things on our server for as long as 30 days</i>

As long as the domains are the same, the 30 day rule doesn't apply--the boxes just can't be out of synch for more than 30 days.

<i>And I actually did look at our earlier conversation regarding this... I thought eth0 would still run after inactivating eth0:5 from active list.</i>

Hmmm...I would think it would. But it might not, if the range syntax is used--in this case all interfaces are in the same configuration file and controlled by the same &quot;ONBOOT&quot; directive. But I didn't think it was possible to define the primary (eth0) with this type of file...only aliases. Which would have meant all but your eth0 were taken down by the change. I'd have to see what your configuration looks like and what changes you made to make a better educated guess about what happened.

--

Check out the forum guidelines!

Sun, 06/07/2009 - 07:01
Blueforce

Hi Joe

I was afraid that this would be yoyr answer, and yes... have tried all I can think of! I actually didn't think the default network would breake when remowing one other IP(interface). The problem with postfix some times picking up another IP than the default I somehow have to resolv. We have access to our ISP's DNS server to for adding, changing DNS settings and entrys.

I'm thinking of installing a box in our office with FC4 and Virtualmin Pro(same license) for cases like this when I'm messing things up like this. I Think I have asked this before if it's ok to do so, the one in our office is only in use and accessible from internet in cases like this. I can easally install it and maybe restore all domains from backupfiles, I have all domains and all wmin config backups.

It's easy for me to change our DNS to point to our private IP instead, and the changes is instant.

Do you think this ok to do, call it our &quot;mirror&quot; server.

Our Server Hall have 24/7 support for those who pay for it, in other words... we are not, and our suport opens in 5 hours so they can manually restart the box.

I'm anyway thinking of doing the mirror thing if that is by license things. I think this is the best way, I could point all domains to our US server but it's running WHM/Cpanel :-( and all customers have to shage their email usernames and so on...

So do you think I should go for it... I can do some usefull work during the night instead of just wating for our Server Hall to open.

Regards,
Leif

Sun, 06/07/2009 - 07:01
Joe
Joe's picture

Hey Leif,

Yes, a hot spare is definitely allowed by our license terms. We're generous to a fault. ;-)

The only rule for hot spares and devel servers: The hot spare must have identical domains to your primary or no domains that live longer than 30 days. Our license manager doesn't actually enforce this limitation at the moment, but it will in about a week, so might as well pretend like it's already a fact.

On the network issue, I'll take this opportunity to remind you of our conversation about changing IPs and how to be safe about it here:

http://www.virtualmin.com/forums/message-view?message_id=64004

But I won't pick on you too much for not being as careful this time. I've done the same thing a half dozen times...thinking I wasn't disabling the only interface that was actually working, or the parent of all other interfaces. ;-)

--

Check out the forum guidelines!

Sun, 06/07/2009 - 07:01
Blueforce

Ok,

And the domains will be exactly the same, and hopfully I don't breake things on our server for as long as 30 days :-)

And I actually did look at our earlier conversation regarding this... I thought eth0 would still run after inactivating eth0:5 from active list.

Will there be some network issues when restoring domains from the backup to our local server, our local sever will get another public IP than the &quot;real&quot; server.

Leif

Sun, 06/07/2009 - 07:01
Joe
Joe's picture

Hey Leif,

<i>And the domains will be exactly the same, and hopfully I don't breake things on our server for as long as 30 days</i>

As long as the domains are the same, the 30 day rule doesn't apply--the boxes just can't be out of synch for more than 30 days.

<i>And I actually did look at our earlier conversation regarding this... I thought eth0 would still run after inactivating eth0:5 from active list.</i>

Hmmm...I would think it would. But it might not, if the range syntax is used--in this case all interfaces are in the same configuration file and controlled by the same &quot;ONBOOT&quot; directive. But I didn't think it was possible to define the primary (eth0) with this type of file...only aliases. Which would have meant all but your eth0 were taken down by the change. I'd have to see what your configuration looks like and what changes you made to make a better educated guess about what happened.

--

Check out the forum guidelines!

Sun, 06/07/2009 - 07:01
Blueforce

Hi Joe,

I have installed our local server but it all took so long that I could call our support to restart the server. Haven't yet try the restore the backups to our hot spare (to be). The strange thing is that the network did not respond with an error when trying to connect with ssh during downtime. puTTY just opened the window with the marker, no time-outs or errors, it just stayed this way untill I closed it manually.

The server did &quot;stop&quot; once more today, 5 hours after the startup this morning in exactly the same way. Our support did check the network card, changed to a new cable just to make sure. I also looked for lost packets and so in ifconfig, everything looked fine to me.

You are very welcome to take a look if you have the time for it. Just let me know if and when you are up for it.

Regards,
Leif

Topic locked