You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Martin Emrich <ma...@empolis.com> on 2016/03/07 15:00:38 UTC

Packet loss on redundant virtual router since <=4.7.1

Hello!

I just upgraded from 4.4.4 to 4.7.1, and upgraded the virtual routers to 4.6.0. Since then, we have some packet loss with redundant virtual routers. If I shut down one of the routers (leaving the other one running), the problem disappears.

I often see messages like "Redundant Virtual Router just switched from Unknown to Master".
I also noticed lots of these messages in the messages-logfile on the master router: "Password server failed with error code 1. Restarting it..."

What could be wrong?

Thanks

Martin Emrich
Senior IT Administrator

Empolis Information Management GmbH | Europaallee 10 | 67657 Kaiserslautern | Germany
Phone +49 631 68037-71 | Fax +49 631 68037-77
martin.emrich@empolis.com<mailto:martin.emrich@empolis.com%0d>

www.empolis.com<http://www.empolis.com/>
Sitz Kaiserslautern  | Amtsgericht Kaiserslautern HRB 31317
Geschäftsführer: Dr. Stefan Wess, Stefan Volland, Dr. Christian Schulmeyer, Dr. Peter Tepassé

SMART INFORMATION MANAGEMENT
Empolis-Lösungen befähigen Unternehmen und Organisationen, die exponentiell wachsende Menge
strukturierter und unstrukturierter Daten zu analysieren, zu interpretieren und automatisiert zu verarbeiten.
Sie nutzen damit ihr Wissenskapital, um unternehmenskritische Geschäftsprozesse zu optimieren.
Entscheider, Mitarbeiter und Kunden erhalten so stets situations- und aufgabengerecht genau die
Information, die für sie relevant ist.
Abonnieren Sie unseren Newsletter<http://newsletter.empolis.com/art_resource.php?sid=si4n.23ctc3r> | Folgen Sie uns auf Facebook<http://www.facebook.com/EmpolisSoftware> | Besuchen Sie uns auf YouTube<http://www.youtube.com/EmpolisSoftware>
[Banner.Experton2016.Signatur]<http://www.empolis.com/>

AW: AW: Packet loss on redundant virtual router since <=4.7.1

Posted by Martin Emrich <ma...@empolis.com>.
Hi!

I did the checks:

- If I shut down the backup via Cloudstack, the packet loss disappears.
- VRRP traffic can be seen both on the master (r-1904-VM) and the backup (r-1905-VM).
- The internal gateway IP is only on the master. (eth0)
- The external IP is on both hosts (eth2):

root@r-1904-VM:/var/log# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 02:00:2f:2b:00:02 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.151/23 brd 192.168.101.255 scope global eth0
    inet 192.168.100.1/32 brd 192.168.101.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 0e:00:a9:fe:01:ce brd ff:ff:ff:ff:ff:ff
    inet 169.254.1.206/16 brd 169.254.255.255 scope global eth1
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 06:be:76:00:01:68 brd ff:ff:ff:ff:ff:ff
    inet 10.32.17.103/20 brd 10.32.31.255 scope global eth2


root@r-1905-VM:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 02:00:18:ba:00:03 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.221/23 brd 192.168.101.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 0e:00:a9:fe:01:f0 brd ff:ff:ff:ff:ff:ff
    inet 169.254.1.240/16 brd 169.254.255.255 scope global eth1
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 06:be:76:00:01:68 brd ff:ff:ff:ff:ff:ff
    inet 10.32.17.103/20 brd 10.32.31.255 scope global eth2

So there's something going on with eth2, both are up, and also have the same MAC address. The keepalived configurations seems to deal only with eth0.

Thanks

Martin 

-----Ursprüngliche Nachricht-----
Von: Remi Bergsma [mailto:RBergsma@schubergphilis.com] 
Gesendet: Montag, 7. März 2016 18:27
An: users@cloudstack.apache.org
Betreff: Re: AW: Packet loss on redundant virtual router since <=4.7.1

The PR mentioned is not related, as it deals with private gateways, which is a vpc feature. 

Changing the MAC address will not help as they will still have the same IP address. One interface is supposed to be down (backup) and one up (master). The first guest tier is used to talk vrrp so make sure they see each other (check with tcpdump to see if the vrrp announce packets make it to the other router. If both are master, they are unable to talk to each other and both assume they are alone. I suspect this is going wrong for you somehow. Please let us know what tcpdump shows. 

There are several integration tests that deal with this so I'm quite sure it should work. 

Regards, Remi 

Sent from my iPhone

> On 07 Mar 2016, at 18:13, Nux! <nu...@li.nux.ro> wrote:
> 
> I am not sure on how to apply this, perhaps Wilder or someone else will be able to advise.
> What I would try to do first is to change the MAC addr on the backup VR to something else and see if that stops the problems.
> 
> --
> Sent from the Delta quadrant using Borg technology!
> 
> Nux!
> www.nux.ro
> 
> ----- Original Message -----
>> From: "Martin Emrich" <ma...@empolis.com>
>> To: users@cloudstack.apache.org
>> Sent: Monday, 7 March, 2016 16:05:02
>> Subject: AW: Packet loss on redundant virtual router since <=4.7.1
> 
>> Hi!
>> 
>> Thanks for the hint! On my test isolated network, currently both 
>> routers have the same configuration on eth2 (the "public" interface), 
>> including the same MAC address, and being up.
>> 
>> So either the VRs do some ARP/ebtables magic to ensure only one 
>> reacts to packets at each given moment, or something is wrong here.
>> 
>> Could that relate to your issue?
>> 
>> I have built ACS RPMs before, what's the easiest way to get a build 
>> with this fix included (just to test whether it applies to us). Note 
>> that we do not use VPC, these are "normal" virtual routers for isolated networks.
>> 
>> Thanks again,
>> 
>> Martin
>> 
>> -----Ursprüngliche Nachricht-----
>> Von: Nux! [mailto:nux@li.nux.ro]
>> Gesendet: Montag, 7. März 2016 16:18
>> An: users@cloudstack.apache.org
>> Betreff: Re: Packet loss on redundant virtual router since <=4.7.1
>> 
>> Can you check nex time this happens if the VRs don't have both the 
>> interface up and same mac addr?
>> Could be this issue https://github.com/apache/cloudstack/pull/1413
>> 
>> HTH
>> 
>> --
>> Sent from the Delta quadrant using Borg technology!
>> 
>> Nux!
>> www.nux.ro
>> 
>> ----- Original Message -----
>>> From: "Martin Emrich" <ma...@empolis.com>
>>> To: users@cloudstack.apache.org
>>> Sent: Monday, 7 March, 2016 14:00:38
>>> Subject: Packet loss on redundant virtual router since <=4.7.1
>> 
>>> Hello!
>>> 
>>> I just upgraded from 4.4.4 to 4.7.1, and upgraded the virtual routers to 4.6.0.
>>> Since then, we have some packet loss with redundant virtual routers.
>>> If I shut down one of the routers (leaving the other one running), 
>>> the problem disappears.
>>> 
>>> I often see messages like "Redundant Virtual Router just switched 
>>> from Unknown to Master".
>>> I also noticed lots of these messages in the messages-logfile on the 
>>> master
>>> router: "Password server failed with error code 1. Restarting it..."
>>> 
>>> What could be wrong?
>>> 
>>> Thanks
>>> 
>>> Martin Emrich
>>> Senior IT Administrator
>>> 
>>> Empolis Information Management GmbH | Europaallee 10 | 67657 
>>> Kaiserslautern | Germany Phone +49 631 68037-71 | Fax +49 631 
>>> 68037-77 
>>> martin.emrich@empolis.com<mailto:martin.emrich@empolis.com%0d>
>>> 
>>> www.empolis.com<http://www.empolis.com/>
>>> Sitz Kaiserslautern  | Amtsgericht Kaiserslautern HRB 31317
>>> Geschäftsführer: Dr. Stefan Wess, Stefan Volland, Dr. Christian Schulmeyer, Dr.
>>> Peter Tepassé
>>> 
>>> SMART INFORMATION MANAGEMENT
>>> Empolis-Lösungen befähigen Unternehmen und Organisationen, die 
>>> exponentiell wachsende Menge strukturierter und unstrukturierter 
>>> Daten zu analysieren, zu interpretieren und automatisiert zu verarbeiten.
>>> Sie nutzen damit ihr Wissenskapital, um unternehmenskritische 
>>> Geschäftsprozesse zu optimieren.
>>> Entscheider, Mitarbeiter und Kunden erhalten so stets situations- 
>>> und aufgabengerecht genau die Information, die für sie relevant ist.
>>> Abonnieren Sie unseren
>>> Newsletter<http://newsletter.empolis.com/art_resource.php?sid=si4n.2
>>> 3c
>>> tc3r> | Folgen Sie uns auf
>>> Facebook<http://www.facebook.com/EmpolisSoftware> | Besuchen Sie uns 
>>> auf YouTube<http://www.youtube.com/EmpolisSoftware>
>>> [Banner.Experton2016.Signatur]<http://www.empolis.com/>

Re: AW: Packet loss on redundant virtual router since <=4.7.1

Posted by Remi Bergsma <RB...@schubergphilis.com>.
The PR mentioned is not related, as it deals with private gateways, which is a vpc feature. 

Changing the MAC address will not help as they will still have the same IP address. One interface is supposed to be down (backup) and one up (master). The first guest tier is used to talk vrrp so make sure they see each other (check with tcpdump to see if the vrrp announce packets make it to the other router. If both are master, they are unable to talk to each other and both assume they are alone. I suspect this is going wrong for you somehow. Please let us know what tcpdump shows. 

There are several integration tests that deal with this so I'm quite sure it should work. 

Regards, Remi 

Sent from my iPhone

> On 07 Mar 2016, at 18:13, Nux! <nu...@li.nux.ro> wrote:
> 
> I am not sure on how to apply this, perhaps Wilder or someone else will be able to advise.
> What I would try to do first is to change the MAC addr on the backup VR to something else and see if that stops the problems.
> 
> --
> Sent from the Delta quadrant using Borg technology!
> 
> Nux!
> www.nux.ro
> 
> ----- Original Message -----
>> From: "Martin Emrich" <ma...@empolis.com>
>> To: users@cloudstack.apache.org
>> Sent: Monday, 7 March, 2016 16:05:02
>> Subject: AW: Packet loss on redundant virtual router since <=4.7.1
> 
>> Hi!
>> 
>> Thanks for the hint! On my test isolated network, currently both routers have
>> the same configuration on eth2 (the "public" interface), including the same MAC
>> address, and being up.
>> 
>> So either the VRs do some ARP/ebtables magic to ensure only one reacts to
>> packets at each given moment, or something is wrong here.
>> 
>> Could that relate to your issue?
>> 
>> I have built ACS RPMs before, what's the easiest way to get a build with this
>> fix included (just to test whether it applies to us). Note that we do not use
>> VPC, these are "normal" virtual routers for isolated networks.
>> 
>> Thanks again,
>> 
>> Martin
>> 
>> -----Ursprüngliche Nachricht-----
>> Von: Nux! [mailto:nux@li.nux.ro]
>> Gesendet: Montag, 7. März 2016 16:18
>> An: users@cloudstack.apache.org
>> Betreff: Re: Packet loss on redundant virtual router since <=4.7.1
>> 
>> Can you check nex time this happens if the VRs don't have both the interface up
>> and same mac addr?
>> Could be this issue https://github.com/apache/cloudstack/pull/1413
>> 
>> HTH
>> 
>> --
>> Sent from the Delta quadrant using Borg technology!
>> 
>> Nux!
>> www.nux.ro
>> 
>> ----- Original Message -----
>>> From: "Martin Emrich" <ma...@empolis.com>
>>> To: users@cloudstack.apache.org
>>> Sent: Monday, 7 March, 2016 14:00:38
>>> Subject: Packet loss on redundant virtual router since <=4.7.1
>> 
>>> Hello!
>>> 
>>> I just upgraded from 4.4.4 to 4.7.1, and upgraded the virtual routers to 4.6.0.
>>> Since then, we have some packet loss with redundant virtual routers.
>>> If I shut down one of the routers (leaving the other one running), the
>>> problem disappears.
>>> 
>>> I often see messages like "Redundant Virtual Router just switched from
>>> Unknown to Master".
>>> I also noticed lots of these messages in the messages-logfile on the
>>> master
>>> router: "Password server failed with error code 1. Restarting it..."
>>> 
>>> What could be wrong?
>>> 
>>> Thanks
>>> 
>>> Martin Emrich
>>> Senior IT Administrator
>>> 
>>> Empolis Information Management GmbH | Europaallee 10 | 67657
>>> Kaiserslautern | Germany Phone +49 631 68037-71 | Fax +49 631 68037-77
>>> martin.emrich@empolis.com<mailto:martin.emrich@empolis.com%0d>
>>> 
>>> www.empolis.com<http://www.empolis.com/>
>>> Sitz Kaiserslautern  | Amtsgericht Kaiserslautern HRB 31317
>>> Geschäftsführer: Dr. Stefan Wess, Stefan Volland, Dr. Christian Schulmeyer, Dr.
>>> Peter Tepassé
>>> 
>>> SMART INFORMATION MANAGEMENT
>>> Empolis-Lösungen befähigen Unternehmen und Organisationen, die
>>> exponentiell wachsende Menge strukturierter und unstrukturierter Daten
>>> zu analysieren, zu interpretieren und automatisiert zu verarbeiten.
>>> Sie nutzen damit ihr Wissenskapital, um unternehmenskritische
>>> Geschäftsprozesse zu optimieren.
>>> Entscheider, Mitarbeiter und Kunden erhalten so stets situations- und
>>> aufgabengerecht genau die Information, die für sie relevant ist.
>>> Abonnieren Sie unseren
>>> Newsletter<http://newsletter.empolis.com/art_resource.php?sid=si4n.23c
>>> tc3r> | Folgen Sie uns auf
>>> Facebook<http://www.facebook.com/EmpolisSoftware> | Besuchen Sie uns
>>> auf YouTube<http://www.youtube.com/EmpolisSoftware>
>>> [Banner.Experton2016.Signatur]<http://www.empolis.com/>

Re: AW: Packet loss on redundant virtual router since <=4.7.1

Posted by Nux! <nu...@li.nux.ro>.
I am not sure on how to apply this, perhaps Wilder or someone else will be able to advise.
What I would try to do first is to change the MAC addr on the backup VR to something else and see if that stops the problems.

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Martin Emrich" <ma...@empolis.com>
> To: users@cloudstack.apache.org
> Sent: Monday, 7 March, 2016 16:05:02
> Subject: AW: Packet loss on redundant virtual router since <=4.7.1

> Hi!
> 
> Thanks for the hint! On my test isolated network, currently both routers have
> the same configuration on eth2 (the "public" interface), including the same MAC
> address, and being up.
> 
> So either the VRs do some ARP/ebtables magic to ensure only one reacts to
> packets at each given moment, or something is wrong here.
> 
> Could that relate to your issue?
> 
> I have built ACS RPMs before, what's the easiest way to get a build with this
> fix included (just to test whether it applies to us). Note that we do not use
> VPC, these are "normal" virtual routers for isolated networks.
> 
> Thanks again,
> 
> Martin
> 
> -----Ursprüngliche Nachricht-----
> Von: Nux! [mailto:nux@li.nux.ro]
> Gesendet: Montag, 7. März 2016 16:18
> An: users@cloudstack.apache.org
> Betreff: Re: Packet loss on redundant virtual router since <=4.7.1
> 
> Can you check nex time this happens if the VRs don't have both the interface up
> and same mac addr?
> Could be this issue https://github.com/apache/cloudstack/pull/1413
> 
> HTH
> 
> --
> Sent from the Delta quadrant using Borg technology!
> 
> Nux!
> www.nux.ro
> 
> ----- Original Message -----
>> From: "Martin Emrich" <ma...@empolis.com>
>> To: users@cloudstack.apache.org
>> Sent: Monday, 7 March, 2016 14:00:38
>> Subject: Packet loss on redundant virtual router since <=4.7.1
> 
>> Hello!
>> 
>> I just upgraded from 4.4.4 to 4.7.1, and upgraded the virtual routers to 4.6.0.
>> Since then, we have some packet loss with redundant virtual routers.
>> If I shut down one of the routers (leaving the other one running), the
>> problem disappears.
>> 
>> I often see messages like "Redundant Virtual Router just switched from
>> Unknown to Master".
>> I also noticed lots of these messages in the messages-logfile on the
>> master
>> router: "Password server failed with error code 1. Restarting it..."
>> 
>> What could be wrong?
>> 
>> Thanks
>> 
>> Martin Emrich
>> Senior IT Administrator
>> 
>> Empolis Information Management GmbH | Europaallee 10 | 67657
>> Kaiserslautern | Germany Phone +49 631 68037-71 | Fax +49 631 68037-77
>> martin.emrich@empolis.com<mailto:martin.emrich@empolis.com%0d>
>> 
>> www.empolis.com<http://www.empolis.com/>
>> Sitz Kaiserslautern  | Amtsgericht Kaiserslautern HRB 31317
>> Geschäftsführer: Dr. Stefan Wess, Stefan Volland, Dr. Christian Schulmeyer, Dr.
>> Peter Tepassé
>> 
>> SMART INFORMATION MANAGEMENT
>> Empolis-Lösungen befähigen Unternehmen und Organisationen, die
>> exponentiell wachsende Menge strukturierter und unstrukturierter Daten
>> zu analysieren, zu interpretieren und automatisiert zu verarbeiten.
>> Sie nutzen damit ihr Wissenskapital, um unternehmenskritische
>> Geschäftsprozesse zu optimieren.
>> Entscheider, Mitarbeiter und Kunden erhalten so stets situations- und
>> aufgabengerecht genau die Information, die für sie relevant ist.
>> Abonnieren Sie unseren
>> Newsletter<http://newsletter.empolis.com/art_resource.php?sid=si4n.23c
>> tc3r> | Folgen Sie uns auf
>> Facebook<http://www.facebook.com/EmpolisSoftware> | Besuchen Sie uns
>> auf YouTube<http://www.youtube.com/EmpolisSoftware>
> > [Banner.Experton2016.Signatur]<http://www.empolis.com/>

AW: Packet loss on redundant virtual router since <=4.7.1

Posted by Martin Emrich <ma...@empolis.com>.
Hi!

Thanks for the hint! On my test isolated network, currently both routers have the same configuration on eth2 (the "public" interface), including the same MAC address, and being up.

So either the VRs do some ARP/ebtables magic to ensure only one reacts to packets at each given moment, or something is wrong here.

Could that relate to your issue?

I have built ACS RPMs before, what's the easiest way to get a build with this fix included (just to test whether it applies to us). Note that we do not use VPC, these are "normal" virtual routers for isolated networks.

Thanks again,

Martin

-----Ursprüngliche Nachricht-----
Von: Nux! [mailto:nux@li.nux.ro] 
Gesendet: Montag, 7. März 2016 16:18
An: users@cloudstack.apache.org
Betreff: Re: Packet loss on redundant virtual router since <=4.7.1

Can you check nex time this happens if the VRs don't have both the interface up and same mac addr?
Could be this issue https://github.com/apache/cloudstack/pull/1413

HTH

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Martin Emrich" <ma...@empolis.com>
> To: users@cloudstack.apache.org
> Sent: Monday, 7 March, 2016 14:00:38
> Subject: Packet loss on redundant virtual router since <=4.7.1

> Hello!
> 
> I just upgraded from 4.4.4 to 4.7.1, and upgraded the virtual routers to 4.6.0.
> Since then, we have some packet loss with redundant virtual routers. 
> If I shut down one of the routers (leaving the other one running), the 
> problem disappears.
> 
> I often see messages like "Redundant Virtual Router just switched from 
> Unknown to Master".
> I also noticed lots of these messages in the messages-logfile on the 
> master
> router: "Password server failed with error code 1. Restarting it..."
> 
> What could be wrong?
> 
> Thanks
> 
> Martin Emrich
> Senior IT Administrator
> 
> Empolis Information Management GmbH | Europaallee 10 | 67657 
> Kaiserslautern | Germany Phone +49 631 68037-71 | Fax +49 631 68037-77 
> martin.emrich@empolis.com<mailto:martin.emrich@empolis.com%0d>
> 
> www.empolis.com<http://www.empolis.com/>
> Sitz Kaiserslautern  | Amtsgericht Kaiserslautern HRB 31317
> Geschäftsführer: Dr. Stefan Wess, Stefan Volland, Dr. Christian Schulmeyer, Dr.
> Peter Tepassé
> 
> SMART INFORMATION MANAGEMENT
> Empolis-Lösungen befähigen Unternehmen und Organisationen, die 
> exponentiell wachsende Menge strukturierter und unstrukturierter Daten 
> zu analysieren, zu interpretieren und automatisiert zu verarbeiten.
> Sie nutzen damit ihr Wissenskapital, um unternehmenskritische 
> Geschäftsprozesse zu optimieren.
> Entscheider, Mitarbeiter und Kunden erhalten so stets situations- und 
> aufgabengerecht genau die Information, die für sie relevant ist.
> Abonnieren Sie unseren
> Newsletter<http://newsletter.empolis.com/art_resource.php?sid=si4n.23c
> tc3r> | Folgen Sie uns auf 
> Facebook<http://www.facebook.com/EmpolisSoftware> | Besuchen Sie uns 
> auf YouTube<http://www.youtube.com/EmpolisSoftware>
> [Banner.Experton2016.Signatur]<http://www.empolis.com/>

Re: Packet loss on redundant virtual router since <=4.7.1

Posted by Nux! <nu...@li.nux.ro>.
Can you check nex time this happens if the VRs don't have both the interface up and same mac addr?
Could be this issue https://github.com/apache/cloudstack/pull/1413

HTH

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Martin Emrich" <ma...@empolis.com>
> To: users@cloudstack.apache.org
> Sent: Monday, 7 March, 2016 14:00:38
> Subject: Packet loss on redundant virtual router since <=4.7.1

> Hello!
> 
> I just upgraded from 4.4.4 to 4.7.1, and upgraded the virtual routers to 4.6.0.
> Since then, we have some packet loss with redundant virtual routers. If I shut
> down one of the routers (leaving the other one running), the problem
> disappears.
> 
> I often see messages like "Redundant Virtual Router just switched from Unknown
> to Master".
> I also noticed lots of these messages in the messages-logfile on the master
> router: "Password server failed with error code 1. Restarting it..."
> 
> What could be wrong?
> 
> Thanks
> 
> Martin Emrich
> Senior IT Administrator
> 
> Empolis Information Management GmbH | Europaallee 10 | 67657 Kaiserslautern |
> Germany
> Phone +49 631 68037-71 | Fax +49 631 68037-77
> martin.emrich@empolis.com<mailto:martin.emrich@empolis.com%0d>
> 
> www.empolis.com<http://www.empolis.com/>
> Sitz Kaiserslautern  | Amtsgericht Kaiserslautern HRB 31317
> Geschäftsführer: Dr. Stefan Wess, Stefan Volland, Dr. Christian Schulmeyer, Dr.
> Peter Tepassé
> 
> SMART INFORMATION MANAGEMENT
> Empolis-Lösungen befähigen Unternehmen und Organisationen, die exponentiell
> wachsende Menge
> strukturierter und unstrukturierter Daten zu analysieren, zu interpretieren und
> automatisiert zu verarbeiten.
> Sie nutzen damit ihr Wissenskapital, um unternehmenskritische Geschäftsprozesse
> zu optimieren.
> Entscheider, Mitarbeiter und Kunden erhalten so stets situations- und
> aufgabengerecht genau die
> Information, die für sie relevant ist.
> Abonnieren Sie unseren
> Newsletter<http://newsletter.empolis.com/art_resource.php?sid=si4n.23ctc3r> |
> Folgen Sie uns auf Facebook<http://www.facebook.com/EmpolisSoftware> | Besuchen
> Sie uns auf YouTube<http://www.youtube.com/EmpolisSoftware>
> [Banner.Experton2016.Signatur]<http://www.empolis.com/>