Problem:
After changing the IP address of an Openstack Kilo installation (and editing all the service endpoints manually), trying to add an Openstack Cloud Provider in ManageIQ (miq) / Cloudforms will return a 503 Service Unavailable error.
Solution:
I will introduce the solution in two parts -- first I will explain how to change the service endpoints in Openstack Kilo from an old to new IP. Second, I will illustrate the deletion of non-existent endpoints from the Keystone database in mariadb which is ultimately causing the 503 error in miq.
Change IP for Existing Openstack Kilo Installation
1. Replace all occurrences of old_ip with new_ip in Openstack config files
My Openstack installation was done with Packstack All-in-one on a physical Fedora 21 server. All the Openstack configuration files can be found under
/etc in subfolders named after component services like
keystone,
neutron,
nova,
cinder, etc. To find all the Openstack config files containing old_ip (which in my case was 192.168.40.198), I changed directories to
/etc and ran the following:
[fedjun@fedrana etc]$ sudo grep -R "192.168.40.198" .
This returned voluminous output that included the following:
./rsync.conf:address = 192.168.40.198
./neutron/neutron.conf:nova_url = http://192.168.40.198:8774/v2
./neutron/neutron.conf:nova_admin_auth_url =http://192.168.40.198:5000/v2.0
./neutron/neutron.conf:auth_uri = http://192.168.40.198:5000/v2.0
./neutron/neutron.conf:identity_uri = http://192.168.40.198:35357
./neutron/neutron.conf:connection = mysql://neutron:2b9473d4f7f24714@192.168.40.1
98/neutron
./neutron/neutron.conf:rabbit_host = 192.168.40.198
./neutron/neutron.conf:rabbit_hosts = 192.168.40.198:5672
./neutron/plugins/openvswitch/ovs_neutron_plugin.ini:local_ip =192.168.40.198
./neutron/metadata_agent.ini:auth_url = http://192.168.40.198:5000/v2.0
./neutron/metadata_agent.ini:nova_metadata_ip = 192.168.40.198
./neutron/api-paste.ini:identity_uri=http://192.168.40.198:35357
./neutron/api-paste.ini:auth_uri=http://192.168.40.198:5000/v2.0
...
My old IP 192.168.40.198 appeared in the following config files (YMMV):
/etc/neutron/neutron.conf
/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini
/etc/neutron/metadata_agent.ini
/etc/neutron/api-paste.ini
/etc/swift/container-server.conf
/etc/swift/account-server.conf
/etc/swift/proxy-server.conf
/etc/libvirt/qemu/instance-00000007.xml
/etc/nova/nova.conf
/etc/ceilometer/ceilometer.conf
/etc/cinder/cinder.conf
/etc/cinder/api-paste.ini
/etc/openstack-dashboard/api-paste.ini
/etc/sysconfig/iptables
/etc/sysconfig/iptables.save
/etc/nagios/nagios_service.cfg
/etc/mongodb/mongodb.conf
/etc/glance/glance-api.conf
/etc/glance/glance-registry.conf
/etc/redis/redis.conf
/etc/httpd/conf.d/15-horizon_vhost.conf
/etc/xinetd.d/rsync.conf
/etc/target/saveconfig.json
You can replace 192.168.40.198 with 192.168.10.168 as follows:
sed -i "s/192.168.40.198/192.168.10.168/g" filename
Note that the old IP address will appear in lots of non-config files as well, such as in various Openstack logs under
/var/log/. You do not need to change the IP's in these log files!
In a post by Brad Pokorny from Symantec about
changing Openstack service IP's, he pipes
grep to
xargs and
sed to replace IP's in conf files:
$ grep -rl '[old IP address]' /etc | xargs sed -i 's/[old IP address]/[new IP address]/g'
grep's
-r option searches recursively (
-R does the same, but unlike
-r it also follows all symlinks) in the search path
grep's
-l option suppresses normal output to stdout and simply prints the name of each file that matches the search pattern.
After editing the IP in all the Openstack and system config files, you are still not done; the old IP's are also saved as service endpoints within the Keystone database in mariadb.
2. Replace all occurrences of old_ip with new_ip in Keystone DB
You can find your mariadb login credentials in your answer file under
$HOME (which should have been automatically generated after running the packstsack or devstack installation script). You can also find the db login credentials within
/etc/keystone/keystone.conf
Log into mariadb:
[fedjun@fedrana ~]$ mysql -ukeystone_admin -p123456789abc
(make sure there isn't a space after the -p flag)
MariaDB [(none)]> use keystone
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
MariaDB [keystone]> show tables;
+------------------------+
| Tables_in_keystone |
+------------------------+
| access_token |
| assignment |
| consumer |
| credential |
| domain |
| endpoint |
| endpoint_group |
| federation_protocol |
| group |
| id_mapping |
| identity_provider |
| idp_remote_ids |
| mapping |
| migrate_version |
| policy |
| policy_association |
| project |
| project_endpoint |
| project_endpoint_group |
| region |
| request_token |
| revocation_event |
| role |
| sensitive_config |
| service |
| service_provider |
| token |
| trust |
| trust_role |
| user |
| user_group_membership |
| whitelisted_config |
+------------------------+
32 rows in set (0.00 sec)
MariaDB [keystone]> show columns from endpoint;
+--------------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+--------------+------+-----+---------+-------+
| id | varchar(64) | NO | PRI | NULL | |
| legacy_endpoint_id | varchar(64) | YES | | NULL | |
| interface | varchar(8) | NO | | NULL | |
| service_id | varchar(64) | NO | MUL | NULL | |
| url | text | NO | | NULL | |
| extra | text | YES | | NULL | |
| enabled | tinyint(1) | NO | | 1 | |
| region_id | varchar(255) | YES | MUL | NULL | |
+--------------------+--------------+------+-----+---------+-------+
8 rows in set (0.00 sec)
MariaDB [keystone]> select * from endpoint
-> ;
+----------------------------------+----------------------------------+----------
-+----------------------------------+--------------------------------------------
------+-------+---------+-----------+
| id | legacy_endpoint_id | interface
| service_id | url
| extra | enabled | region_id |
+----------------------------------+----------------------------------+----------
-+----------------------------------+--------------------------------------------
------+-------+---------+-----------+
| 0824196fa17c446ca255e2978b6541f0 | 27e475394a484791951b539e3a0f933e | internal
| 3c15bfa190cf4ea4af059978a71615dd | http://192.168.40.198:8080/v1/AUTH_%(tenant_id)s | {} | 1 | RegionOne |
| 0d871e7d98124e78a452e192636386d7 | 843a20e3a0b84895ba10425f5631580c | admin | c58ff58e9b2249798402afffe880acd3 | http://127.0.0.1:8774/v3 | {} | 1 | RegionOne |
| 2235601f13404931858d15d919bc0c5f | e7149dd44359447fbbd85d0bb502694c | admin | a2cd703896714905afc15ab7e909902d | http://192.168.40.198:9292 | {} | 1 | RegionOne |
| 2c5edf0ae12a47448a92259d0db26ce7 | df679e86729d48fda2d9be9f5b960171 | admin | 8fcfa027b51c4e40aeee09244d6400b7 | http://192.168.40.198:8773/services/Admin | {} | 1 | RegionOne |
| 2e2cbf33223f4c79b88442e2df3f7b6b | ad6f8e52ff1f4708b64dee23098021bd | public | ed420763fd2a4ff3b9a858557bb4b5e0 | http://192.168.40.198:8774/v2/%(tenant_id)s | {} | 1 | RegionOne |
| 2ebacc0d9f0a47b79a8d003a2097756c | 45a8c7dda03344e58f4921aa2bbad62e | admin | 33436af568cf4236936f4b6645a676d4 | http://192.168.40.198:8776/v1/%(tenant_id)s | {} | 1 | RegionOne |
...
+----------------------------------+----------------------------------+-----------+----------------------------------+--------------------------------------------------+-------+---------+-----------+
33 rows in set (0.01 sec)
Now replace all occurrences of old_ip with new_ip:
MariaDB [keystone]> update endpoint set url = replace(url, '192.168.40.198', '192.168.10.168');
Query OK, 30 rows affected (0.04 sec)
Rows matched: 33 Changed: 30 Warnings: 0
Now you must restart the keystone service:
[fedjun@fedrana ~]$ sudo systemctl restart openstack-keystone
[fedjun@fedrana ~]$ systemctl status openstack-keystone
● openstack-keystone.service - OpenStack Identity Service (code-named Keystone)
Loaded: loaded (/usr/lib/systemd/system/openstack-keystone.service; enabled)
Active: active (running) since Thu 2016-01-14 13:58:13 KST; 9s ago
Main PID: 13518 (keystone-all)
CGroup: /system.slice/openstack-keystone.service
├─13518 /usr/bin/python /usr/bin/keystone-all
├─13527 /usr/bin/python /usr/bin/keystone-all
├─13528 /usr/bin/python /usr/bin/keystone-all
├─13529 /usr/bin/python /usr/bin/keystone-all
├─13530 /usr/bin/python /usr/bin/keystone-all
├─13531 /usr/bin/python /usr/bin/keystone-all
└─13532 /usr/bin/python /usr/bin/keystone-all
Horizon UI should now appear and work with your new IP. Note that you will also have to edit your external network (and floating IP ranges) and change the gateway in your virtual router in Neutron, as they will still be using the old IP.
Remove unused endpoints from Keystone DB to solve 503 error in miq
My packstack install of Openstack Kilo does not include Swift Object Storage, but when adding an Openstack Cloud Provider to miq, one of the logs within the Cloudforms / miq appliance indicated that no endpoint for Swift on port 8080 could be found. When I navigated to the Openstack IP on port 8080 I got the following:
By contrast, navigating by browser to any other port corresponding to an Openstack service endpoint returns JSON or XML. To enable verbose debugging in miq / Cloudforms, follow the instructions from the link below:
http://talk.manageiq.org/t/get-error-when-adding-a-new-provider/961/4
In the ManageIQ UI go to configure/configuration/, Select the appliance and go to Advanced tab. Find the following line and make the change below.
level_fog: info
to
level_fog: debug
Once this is done, wait a few minutes for the changes to take effect in miq / Cloudforms and then log into your miq appliance and navigate to
/var/www/miq/vmdb/log/fog.log
Note that the password for the miq appliance is
root:smartvm whereas the default
user:pass for the miq web interface is
admin:smartvm.
Looking through the logs I noticed that the 503 error only occurred when a request was made on port 8080:
Output within
fog.log indicates that port 8080 is used for the swift service endpoint:
{"endpoints"=>
[{"adminURL"=>"http://192.168.10.168:8080",
"region"=>"RegionOne",
"internalURL"=>
"http://192.168.10.168:8080/v1/AUTH_1e6f3fe16a7b4cb7ad2feb797172a7a0",
"id"=>"0824196fa17c446ca255e2978b6541f0",
"publicURL"=>
"http://192.168.10.168:8080/v1/AUTH_1e6f3fe16a7b4cb7ad2feb797172a7a0"}],
"endpoints_links"=>[],
"type"=>"object-store",
"name"=>"swift"},
I confirmed that port 8080 is used by swift from the following Openstack reference:
http://docs.openstack.org/kilo/config-reference/content/firewalls-default-ports.html
The link above states that port 8080 is the HTTP alternate for OpenStack Object Storage (swift) service. The problem is that I never installed Swift in my Openstack Kilo deployment, but Keystone DB is telling miq that the endpoint for swift is at
:8080. Since this endpoint doesn't exist, miq fails when trying to add the Openstack Cloud Provider. The weird thing is, before I changed my Openstack IP, I could create an Openstack Cloud Provider in miq without any problems.
The solution for the problem of non-existent endpoints is to go into the Keystone DB and delete all rows containing port 8080 (for swift) from the table
endpoints:
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| keystone |
| test |
+--------------------+
3 rows in set (0.05 sec)
MariaDB [(none)]> use keystone
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [keystone]> select url from endpoint;
+--------------------------------------------------+
| url |
+--------------------------------------------------+
| http://192.168.10.168:8080/v1/AUTH_%(tenant_id)s |
| http://127.0.0.1:8774/v3 |
| http://192.168.10.168:9292 |
| http://192.168.10.168:8773/services/Admin |
| http://192.168.10.168:8774/v2/%(tenant_id)s |
| http://192.168.10.168:8776/v1/%(tenant_id)s |
| http://192.168.10.168:8776/v2/%(tenant_id)s |
| http://192.168.10.168:9292 |
| http://192.168.10.168:8774/v2/%(tenant_id)s |
| http://192.168.10.168:8774/v2/%(tenant_id)s |
| http://192.168.10.168:8776/v1/%(tenant_id)s |
| http://192.168.10.168:8080/v1/AUTH_%(tenant_id)s |
| http://192.168.10.168:8080 |
| http://192.168.10.168:8777 |
| http://192.168.10.168:8080 |
| http://192.168.10.168:35357/v2.0 |
| http://192.168.10.168:9696 |
| http://192.168.10.168:8776/v2/%(tenant_id)s |
| http://192.168.10.168:8776/v2/%(tenant_id)s |
| http://192.168.10.168:9292 |
| http://192.168.10.168:8080 |
| http://192.168.10.168:8773/services/Cloud |
| http://192.168.10.168:5000/v2.0 |
| http://192.168.10.168:8773/services/Cloud |
| http://192.168.10.168:9696 |
| http://192.168.10.168:8776/v1/%(tenant_id)s |
| http://192.168.10.168:5000/v2.0 |
| http://192.168.10.168:8777 |
| http://127.0.0.1:8774/v3 |
| http://192.168.10.168:8080 |
| http://127.0.0.1:8774/v3 |
| http://192.168.10.168:9696 |
| http://192.168.10.168:8777 |
+--------------------------------------------------+
33 rows in set (0.00 sec)
I can see a variety of endpoints above, but just want to get rid of all the rows containing port 8080. In mysql/mariadb it is possible to match text patterns as below:
MariaDB [keystone]> select * from endpoint where url like '%:8080%';
+----------------------------------+----------------------------------+-----------+-------
---------------------------+--------------------------------------------------+-------+---
------+-----------+
| id | legacy_endpoint_id | interface | servic
e_id | url | extra | en
abled | region_id |
+----------------------------------+----------------------------------+-----------+-------
---------------------------+--------------------------------------------------+-------+---
------+-----------+
| 0824196fa17c446ca255e2978b6541f0 | 27e475394a484791951b539e3a0f933e | internal | 3c15bf
a190cf4ea4af059978a71615dd | http://192.168.10.168:8080/v1/AUTH_%(tenant_id)s | {} |
1 | RegionOne |
| 4a925e2860da4625b16ef3efd3ef1147 | 27e475394a484791951b539e3a0f933e | public | 3c15bf
a190cf4ea4af059978a71615dd | http://192.168.10.168:8080/v1/AUTH_%(tenant_id)s | {} | 1 | RegionOne |
| 4cda842dd58b4cfb91ec925f135be45d | 27e475394a484791951b539e3a0f933e | admin | 3c15bfa190cf4ea4af059978a71615dd | http://192.168.10.168:8080 | {} | 1 | RegionOne |
| 5af0b27d3efe4a2d84bbf0b0b79a0642 | 06ec34b3c27a4164beb4c142bb4a890b | internal | 06da29032bb74154834ce004621e1707 | http://192.168.10.168:8080 | {} | 1 | RegionOne |
| ab9bf978255b496b9fc62d46b90ee211 | 06ec34b3c27a4164beb4c142bb4a890b | public | 06da29032bb74154834ce004621e1707 | http://192.168.10.168:8080 | {} | 1 | RegionOne |
| e29c2509559e48dbabe4d295f2877e43 | 06ec34b3c27a4164beb4c142bb4a890b | admin | 06da29032bb74154834ce004621e1707 | http://192.168.10.168:8080 | {} | 1 | RegionOne |
+----------------------------------+----------------------------------+-----------+----------------------------------+--------------------------------------------------+-------+---------+-----------+
6 rows in set (0.00 sec)
Now I can delete these matches from the table
endpoint like this:
MariaDB [keystone]> delete from endpoint where url like '%:8080%';
Query OK, 6 rows affected (0.21 sec)
After restarting keystone service, adding an Openstack Cloud Provider in miq works fine!
