Cloudstack平台安全研究
时间：2015-01-11 来源：服务器在线 投稿：泡泡 点击：次
面向Cloudstack4.2 Security Group XenServer 6.x ,对于KVM原理一样，需要和Agent通信，调用security_group.py。
安全组是AWS里的一种技术，用于隔离多租户虚拟机，Cloudstack也支持安全组，3.0.x版本只支持Basic Zone的安全组，4.1以后引入Advance Shared Zone的安全组，目前安全组支持的Hypervisor为KVM 和Xen系列，对于VMware可以使用PVLAN也可以使用vShield实现，需要独立开发。
首先要创建带安全组的Shared网络服务，创建Zone，创建VM时，选择新创建的安全组或默认的。如果不选择安全组，会使用默认的安全组。默认情况下，安全组会将VM的Deny All Ingress ，Allow ALL Egress.
对于Xenserver 如果要使用安全组，需要将网络从OVS改为Bridge。在xenserver机器执行
#cat /etc/xensource/network.confopenvswitch
执行改为bridge或者手动改为bridge,并重新启动主机。
# xe-switch-network-backend bridge另外需要确认以下这些值，如下表示OK。
#sysctl -a|grep bridgenet.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-arptables = 1
如果不是，修改
/etc/sysctl.conf#sysctl -p
CitrixResourceBase.java 是负责和XenServer交互的底层资源类。在VM启动时，如果启用安全组，将建立默认规则(Ingress Allow,Egress Deny).
方法为 execute(StartCommand cmd),里面代码片段：
if (_canBridgeFirewall) {String result = null;
if (vmSpec.getType() != VirtualMachine.Type.User) {
NicTO[] nics = vmSpec.getNics();
boolean secGrpEnabled = false;
for (NicTO nic : nics) {
if (nic.isSecurityGroupEnabled()
|| (nic.getIsolationUri() != null && nic.getIsolationUri().getScheme()
.equalsIgnoreCase(IsolationType.Ec2.toString()))) {
secGrpEnabled = true;
break;
}
}
if (secGrpEnabled) {
result = callHostPlugin(conn, "vmops", "default_network_rules_systemvm", "vmName", vmName);
if (result == null || result.isEmpty() || !Boolean.parseBoolean(result)) {
s_logger.warn("Failed to program default network rules for " + vmName);
} else {
s_logger.info("Programmed default network rules for " + vmName);
}
}
} else {
// For user vm, program the rules for each nic if the isolation
// uri scheme is ec2
NicTO[] nics = vmSpec.getNics();
for (NicTO nic : nics) {
if (nic.isSecurityGroupEnabled() || nic.getIsolationUri() != null
&& nic.getIsolationUri().getScheme().equalsIgnoreCase(IsolationType.Ec2.toString())) {
List<String> nicSecIps = nic.getNicSecIps();
String secIpsStr;
StringBuilder sb = new StringBuilder();
if (nicSecIps != null) {
for (String ip : nicSecIps) {
sb.append(ip).append(":");
}
secIpsStr = sb.toString();
} else {
secIpsStr = "0:";
}
result = callHostPlugin(conn, "vmops", "default_network_rules", "vmName", vmName, "vmIP",
nic.getIp(), "vmMAC", nic.getMac(), "vmID", Long.toString(vmSpec.getId()), "secIps",
secIpsStr);
if (result == null || result.isEmpty() || !Boolean.parseBoolean(result)) {
s_logger.warn("Failed to program default network rules for " + vmName + " on nic with ip:"
+ nic.getIp() + " mac:" + nic.getMac());
} else {
s_logger.info("Programmed default network rules for " + vmName + " on nic with ip:"
+ nic.getIp() + " mac:" + nic.getMac());
}
}
}
}
}
该方法将判断主机网络是否为bridge，如果不是就不会建立安全组规则。将遍历VM所有网卡，判断是否启用安全组隔离类型为EC2，即安全组隔离。
对于System VM（vrouter,ssvm,cpvm），要单独处理，如果是System VM,会调用主机plugin vmops里的方法default_network_rules_systemvm 参数为VM名字，如果是User VM,会调用default_network_rules，参数为 vmName, vmIP ，vmMAC,， secIps，secIps表示网卡的第二个IP地址，cloudstack支持一个网卡多个IP地址。
主机Plugin vmops 位于cloudstack 代码树中cloudstack/scripts/vm/hypervisor/xenserver,在Xenserver主机/etc/xapi.d/plugins/
Cloudstack对于Xenserver和KVM的安全组主要使用Iptables，ebtable ，ipset，arptables实现。
System VMdef default_network_rules_systemvm(session, args):
......
vmchain = chain_name(vm_name)
.......
delete_rules_for_vm_in_bridge_firewall_chain(vm_name)
#建立iptables chain
try:
util.pread2(['iptables', '-N', vmchain])
except:
util.pread2(['iptables', '-F', vmchain])
#允许所有出流量
allow_egress_traffic(session)
#遍历vif
for vif in vifs:
try:
util.pread2(['iptables', '-A', 'BRIDGE-FIREWALL', '-m', 'physdev', '--physdev-is-bridged', '--physdev-out', vif, '-j', vmchain])
util.pread2(['iptables', '-I', 'BRIDGE-FIREWALL', '2', '-m', 'physdev', '--physdev-is-bridged', '--physdev-in', vif, '-j', vmchain])
util.pread2(['iptables', '-I', vmchain, '-m', 'physdev', '--physdev-is-bridged', '--physdev-in', vif, '-j', 'RETURN'])
except:
util.SMlog("Failed to program default rules")
return 'false'
util.pread2(['iptables', '-A', vmchain, '-j', 'ACCEPT'])
return 'true'
SSVM Chain规则
Chain s-6-VM (8 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif18.2 --physdev-is-bridged
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif18.0 --physdev-is-bridged
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif18.1 --physdev-is-bridged
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif18.3 --physdev-is-bridged
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
CPVM Chain规则
Chain v-2-VM (6 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif17.2 --physdev-is-bridged
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif17.0 --physdev-is-bridged
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif17.1 --physdev-is-bridged
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
VRouter Chain规则
Chain r-4-VM (4 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif19.0 --physdev-is-bridged
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif19.1 --physdev-is-bridged
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
User VM
def default_network_rules(session, args):
vm_name = args.pop('vmName')
vm_ip = args.pop('vmIP')
vm_id = args.pop('vmID')#
vm_mac = args.pop('vmMAC')
sec_ips = args.pop("secIps")
action = "-A"
try:
vm = session.xenapi.VM.get_by_name_label(vm_name)
if len(vm) != 1:
util.SMlog("### Failed to get record for vm " + vm_name)
return 'false'
vm_rec = session.xenapi.VM.get_record(vm[0])
domid = vm_rec.get('domid')
except:
util.SMlog("### Failed to get domid for vm " + vm_name)
return 'false'
if domid == '-1':
util.SMlog("### Failed to get domid for vm (-1): " + vm_name)
return 'false'
vif = "vif" + domid + ".0"
tap = "tap" + domid + ".0"
vifs = [vif]
try:
util.pread2(['ifconfig', tap])
vifs.append(tap)
except:
pass
delete_rules_for_vm_in_bridge_firewall_chain(vm_name)
vmchain = chain_name(vm_name)
vmchain_egress = egress_chain_name(vm_name) #EGRESS
vmchain_default = chain_name_def(vm_name) #INGRESS
destroy_ebtables_rules(vmchain)
#建立三个chain: i-x-y-VM ,i-x-y-VM-eg,i-x-y-VM-def
try:
util.pread2(['iptables', '-N', vmchain])
except:
util.pread2(['iptables', '-F', vmchain])
try:
util.pread2(['iptables', '-N', vmchain_egress])
except:
util.pread2(['iptables', '-F', vmchain_egress])
try:
util.pread2(['iptables', '-N', vmchain_default])
except:
util.pread2(['iptables', '-F', vmchain_default])
vmipset = vm_name
#create ipset and add vm ips to that ip set
if create_ipset_forvm(vmipset) == False:
util.SMlog(" failed to create ipset for rule " + str(tokens))
return 'false'
#add primary nic ip to ipset
if add_to_ipset(vmipset, [vm_ip], action ) == False:
util.SMlog(" failed to add vm " + vm_ip + " ip to set ")
return 'false'
#add secodnary nic ips to ipset
secIpSet = "1"
ips = sec_ips.split(':')
ips.pop()
if ips[0] == "0":
secIpSet = "0";
if secIpSet == "1":
util.SMlog("Adding ipset for secondary ips")
add_to_ipset(vmipset, ips, action)
if write_secip_log_for_vm(vm_name, sec_ips, vm_id) == False:
util.SMlog("Failed to log default network rules, ignoring")
keyword = '--' + get_ipset_keyword()
try:
for v in vifs:
util.pread2(['iptables', '-A', 'BRIDGE-FIREWALL', '-m', 'physdev', '--physdev-is-bridged', '--physdev-out', v, '-j', vmchain_default])
util.pread2(['iptables', '-I', 'BRIDGE-FIREWALL', '2', '-m', 'physdev', '--physdev-is-bridged', '--physdev-in', v, '-j', vmchain_default])
#don't let vm spoof its ip address
for v in vifs:
#允许VM的DNS请求
util.pread2(['iptables', '-A', vmchain_default, '-m', 'physdev', '--physdev-is-bridged', '--physdev-in', v, '-m', 'set', keyword, vmipset, 'src', '-p', 'udp', '--dport', '53', '-j', 'RETURN'])
util.pread2(['iptables', '-A', vmchain_default, '-m', 'physdev', '--physdev-is-bridged', '--physdev-in', v, '-m', 'set', '!', keyword, vmipset, 'src', '-j', 'DROP'])
util.pread2(['iptables', '-A', vmchain_default, '-m', 'physdev', '--physdev-is-bridged', '--physdev-out', v, '-m', 'set', '!', keyword, vmipset, 'dst', '-j', 'DROP'])
util.pread2(['iptables', '-A', vmchain_default, '-m', 'physdev', '--physdev-is-bridged', '--physdev-in', v, '-m', 'set', keyword, vmipset, 'src', '-j', vmchain_egress])
util.pread2(['iptables', '-A', vmchain_default, '-m', 'physdev', '--physdev-is-bridged', '--physdev-out', v, '-j', vmchain])
except:
util.SMlog("Failed to program default rules for vm " + vm_name)
return 'false'
default_arp_antispoof(vmchain, vifs, vm_ip, vm_mac)
#add default arp rules for secondary ips;
if secIpSet == "1":
util.SMlog("Adding arp rules for sec ip")
arp_rules_vmip(vmchain, vifs, ips, vm_mac, action)
default_ebtables_antispoof_rules(vmchain, vifs, vm_ip, vm_mac)
if write_rule_log__vm(vm_name, vm_id, vm_ip, domid, '_initial_', '-1', vm_mac) == False:
util.SMlog("Failed to log default network rules, ignoring")
util.SMlog("Programmed default rules for vm " + vm_name)
return 'true'
用户VM chain规则
Chain i-2-7-VM (1 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0
Chain i-2-7-VM-eg (1 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0
Chain i-2-7-def (2 references)
target prot opt source destination
RETURN udp -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif21.0 --physdev-is-bridged set i-2-7-VM src udp dpt:53
DROP all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif21.0 --physdev-is-bridged !set i-2-7-VM src
DROP all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif21.0 --physdev-is-bridged !set i-2-7-VM dst
i-2-7-VM-eg all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif21.0 --physdev-is-bridged set i-2-7-VM src
i-2-7-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif21.0 --physdev-is-bridged
对于User VM，会创建ebtables和ipset,arptable。并且允许DHCP 67/68端口,否则用户VM将无法自动获得IP。
ebtablesBridge chain: FORWARD, entries: 5, policy: ACCEPT
-j DEFAULT_EBTABLES
-i vif21.0 -j i-2-7-VM #入接口
-o vif21.0 -j i-2-7-VM #出接口
Bridge chain: OUTPUT, entries: 0, policy: ACCEPT
Bridge chain: DEFAULT_EBTABLES, entries: 12, policy: ACCEPT
#DHCP的广播目的地址是255.255.255.255，源地址是0.0.0.0
-p IPv4 --ip-dst 255.255.255.255 --ip-proto udp --ip-dport 67 -j ACCEPT
-p IPv4 --ip-dst 255.255.255.255 --ip-proto udp --ip-dport 68 -j ACCEPT
-p ARP --arp-op Request -j ACCEPT
-p ARP --arp-op Reply -j ACCEPT
-p IPv4 -d Broadcast -j DROP
-p IPv4 -d Multicast -j DROP
-p IPv4 --ip-dst 255.255.255.255 -j DROP
-p IPv4 --ip-dst 224.0.0.0/4 -j DROP
-p IPv4 -j RETURN
-p IPv6 -j DROP
-p 802_1Q -j DROP
-j DROP
Bridge chain: i-2-7-VM, entries: 2, policy: ACCEPT
# fake dhcp and snooping of dhcp requests drop
-p IPv4 -i vif21.0 --ip-proto udp --ip-dport 68 -j DROP
-p IPv4 -o vif21.0 --ip-proto udp --ip-dport 67 -j DROP
IPSet
ipset是以VM名字命名的，membes是VM的IP地址。
Name: i-2-7-VMType: iphash
References: 4
Header: hashsize: 1024 probes: 8 resize: 50
Members:
192.168.2xx.***
对于target 为RETURN 的会交给上级chain 处理。
Chain (1 references)target prot opt source destination
BRIDGE-DEFAULT-FIREWALL all -- 0.0.0.0/0 0.0.0.0/0
i-2-7-def all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif21.0 --physdev-is-bridged
r-4-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif19.0 --physdev-is-bridged
r-4-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif19.1 --physdev-is-bridged
s-6-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif18.2 --physdev-is-bridged
s-6-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif18.0 --physdev-is-bridged
s-6-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif18.1 --physdev-is-bridged
s-6-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif18.3 --physdev-is-bridged
v-2-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif17.2 --physdev-is-bridged
v-2-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif17.0 --physdev-is-bridged
v-2-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vif17.1 --physdev-is-bridged
v-2-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif17.1 --physdev-is-bridged
v-2-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif17.0 --physdev-is-bridged
v-2-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif17.2 --physdev-is-bridged
s-6-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif18.3 --physdev-is-bridged
s-6-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif18.1 --physdev-is-bridged
s-6-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif18.0 --physdev-is-bridged
s-6-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif18.2 --physdev-is-bridged
r-4-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif19.1 --physdev-is-bridged
r-4-VM all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif19.0 --physdev-is-bridged
i-2-7-def all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out vif21.0 --physdev-is-bridged
默认DHCP的67，68端口允许。
Chain FORWARD (policy ACCEPT)target prot opt source destination
BRIDGE-FIREWALL all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-is-bridged
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out eth1 --physdev-is-bridged
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out eth0 --physdev-is-bridged
DROP all -- 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain BRIDGE-DEFAULT-FIREWALL (1 references)
target prot opt source destination
如果您的问题仍未解决，还可以加入服务器在线技术交流QQ群:8017413寻求帮助。
相关内容
最新热点内容
推荐内容
